basic scraping question

[Stanley]

I'm using file_get_contents to scrape a few hundred pages, but the load time for each page is significant due to images and flash. What would be a more efficient method of getting that content without downloading the extraneous junk?

 

[nickycakes]

Well, php won't download any of that crap, only the file you're telling it to download.

Unless you mean you want to parse out all the embedded objects and image tags in the file to reduce load time after it's already been scraped and you open it with your browser, I don't think there's much you can do to speed up file_get_contents calls without using a faster server.

 

[bubbles]

I don't know if any of these are more efficient, (although I think I remember reading that just about anything is more efficient than file_get_contents()) you could always use these instead:
curl
file()
fope()

 

[Stanley]

I tested file_get_contents vs curl()

Test 1: 5.6382260322571
Test 2: 4.8649210929871

Test 1: 18.499569177628
Test 2: 16.000211000443

Test 1: 21.442551136017
Test 2: 15.969302177429

Test 1 is file_get_contents
Test 2 is curl()

 

[nickycakes]

would you mind testing:
file_get_contents()
vs.
implode("",file());

 

[drloveless]

i would use curl over file_get_contents any day.

 

[Supergeek]

wget in a shell script.