yahoo blocking scraping?

Status
Not open for further replies.
Webferret

Webferret

Banned
Jun 18, 2007
1,257
10
0
anyone else having problems with scraping yahoo news feeds in php?

suddenly I'm starting to get

Warning: fopen() [function.fopen]: HTTP request failed! HTTP/1.1 999 Unable to process request at this time -- error 999 in etc...

up till a few days ago it worked fine... doing all the usual things, random user agent etc. maybe yahoozier has banned my site's ip address? Mmmm...
 


rotate proxies. thats what I do, the other option is use yahoo.co.uk it has no filter
 
yahoo uk same problem. looks like ip ban. been in place over a day now. So.... move the site or knock up a proxy shuffle. anyone got a dynamically updating list of proxy servers???
 
tor is interesting, and a very worthwhile project, but can it be integrasted onto a server running php scrapes? Isn't it just client software? How could it allow my server to anonymise requests for pages from yahoozer?
 
I've done it. Granted, it was on an Ubuntu server so all I did was "Sudo apt-get tor" (I think the package name was tor.)

Then setting curl to use the proxy at the Tor port.
 
Webferret said:
tor is interesting, and a very worthwhile project, but can it be integrasted onto a server running php scrapes? Isn't it just client software? How could it allow my server to anonymise requests for pages from yahoozer?
Click to expand...

Tor works fine via PHP & w/Yahoo. Go into the Design forum and search. Running 24x7 w/no 999 issues.
 
learn something new every day. Ta! Wonder what happened to that poor danish sod running a tor node who got arrested for 'child porn'.
 
OK, here's how I fixed it.

in the function that does th efopen, if it fails, fall thru to a second function (cribbed from php.net).

I'm 'improved' this function so what it does is whip off to a well know site offering a list of currently working proxy ips, scrape them, choose 1 at random, then try the original page again using this instead of fopen.

Adds another call to another site (to get the proxy ips), but I cache it once a day if it's called, and so far seems to work like a charm
 
Status
Not open for further replies.
Top Bottom