Let's say that someone wants me to process a list of 125,000 domains and return the list with each domain's page rank and alexa rank. Let's say that I'm writing a script that can do this. How much should I charge for this service? I'm thinking price per thousand domains.
How much should I charge for this?
- Thread starter alexb
- Start date
- Status
Big ?: How long is the turnaround time?
You need to stay under the radar. It will take your server 3 days to cycle through 125k pages if doing 1 scrape every 2 seconds. I'm thinking you'll get your IP blocked every few hours if that's the case, but I can't tell you what the limit is because I honestly don't know.
Once your IP gets blocked by either service, there will be some additional hard costs involved on your part (new IP|domain|server).
If you have the PR & alexa scraping scripts already I'd probably go $3 per 1000 turned around in 7 days. I certainly would not do this for $3 if it's just 1000 sites but since you have a big number to work with...
Hooking this up wouldn't take me an hour or so worth of work, but if you don't have the functions in place you are looking at a much more labor intensive project.
You need to stay under the radar. It will take your server 3 days to cycle through 125k pages if doing 1 scrape every 2 seconds. I'm thinking you'll get your IP blocked every few hours if that's the case, but I can't tell you what the limit is because I honestly don't know.
Once your IP gets blocked by either service, there will be some additional hard costs involved on your part (new IP|domain|server).
If you have the PR & alexa scraping scripts already I'd probably go $3 per 1000 turned around in 7 days. I certainly would not do this for $3 if it's just 1000 sites but since you have a big number to work with...
Hooking this up wouldn't take me an hour or so worth of work, but if you don't have the functions in place you are looking at a much more labor intensive project.
sign up at 20 freehosts that allow PHP, and add a script to check pagerank onto each one.
I've got about 10 "satellite" domains on cheap ass servers because I don't like to waste time with free hosts (some don't allow curl, some are already blocked, some are slow as piss, etc) $5/year hosts are not much better but not killed in the same fashion that free ones are.
I put a bunch of scraping scripts on each of them and just call randomly. It works pretty well for a $100 / year setup. This allows me to scrape whatever I want to and never expose my money servers.
Quick question, when scraping google (SERPs or PR) what's your experience as to how long in between queries you need to delay in order to not get banned? I've never had a problem, but I've also never hit 125k in a few hours.
It depends. a 2 second seperation I've rarely(or never I think) gotten banned. 1 second seperation I get banned, but after a decent number of queries.
Anything less than that and it's quite fast.
- Status