genetic

I am coding a scraping API. It's for a service that I am going to launch. Development has been ongoing for 3 days so far.

Although the API front-end is not started yet, the back-end (processing) is in a late Alpha stage now, and is capable of handling scrapes for the source of any URL that doesn't require a POST. In simple terms it cannot scrape data from a page that requires form input (yet).

I would like to test what I have implemented so far with real scraping requests, as I'm not sure if I am going too easy on myself with the tests I have come up with so far.

If anybody has any sites that they would like some data scraped from, I can do this for you (FREE!) as part of my testing.

The details I require are as follows -


One thing I will mention is that this is not an unlimited offer, in that I am not necessarily offering it to unlimited people, nor am I offering to scrape an unlimited amount of data. I think a reasonable amount of URLs to scrape is 500. Remember this is a fully hosted service.

Any questions, or requests for scrapes, just reply in this thread. If anything you want to scrape is non-public, I'm sure you know where the PM button is!

dchuk

what'd ya build it with?

__________________

Bofu2U

So wait, is this like a .. I give you a URL, and it scrapes the entire site or... what?

__________________

xpathfucker

I love scrapping. It's like licking your cousins tits. You know it's wrong but it really tastes so sweet... Just can't help myself.

Anyway, can you scrape images too or just simple text?

Jake232

Probably Anarchid

genetic

It's all in PHP. I'm sure many will feel that it isn't the best language to do something like this in, but it's what I know best, and (so far) it seems to be working well.

You got any sites for me to scrape dchuk? :P

dchuk

nah, I'm good, I have my own scrapers when I need them (not knocking what you've done, just leaving a spot open for someone else who needs scraping done)

I wrote this in ruby, might want to play with it, seems like it would be useful for what you're doing: http://github.com/dchuk/Arachnid

__________________

genetic

Several replies since I replied to dchuk.

@Bofu2U This is a lot more "logical" than that. For example you want all the "First Name" entries from a site. You give me the URL of page one of the listing, and the script grabs every "First Name" entry throughout listing, including entries on any pagination. More complicated scrapes are possible, but this is the basic idea.

@xpathfucker "it's wrong but it really tastes so sweet..." <-- THIS. The script can scrape text, but if it is scraping images it can upload them to a specified FTP sever or return their original URLs.

@Jake232 Eh?

Bofu2U

What if I give you one page with an example of the structure (and what I want, like First name) and tell you I want that information from every page on the entire site?

__________________

genetic

Ahh, it seems that Jake232 was talking about Arachnid by dchuk.

@dchuk Yeah, I appreciate that a lot of people will have their own solutions for any scraping requirements that they may have, but at the same time there are many who do not have access to this kind of resource. This is the gap(?) in the market that I am looking to fill with this service. Thanks for the sharing the code for Arachnid, I will have to have a look, and at the very least understand the approach you have taken. Hopefully I can learn something from your methods!

xpathfucker

Ok cool. May I suggest you include a way to scrape stuff that require post requests (which is quite easy anyway), cause that is where some powerfull things can be made with PHP.

No urls to give u, but u sure have a nice offer. Have fun scrapping.

genetic

This is exactly the objective I am working to achieve. Nice and automated, define the URL to scrape, and then grab the required data from the entire site so reliably that it can be handled by other scripts or even (spun? [and]) used as content.

Bofu2U

That works. If you want a nice stress test and have plenty of proxies let me know.

__________________

eliquid

if your using php, are you using curl, raw sockets, or fopen?

if curl, are you using an asyn/non blocking version of it? how about polling/que?

just curious, I've been testing up a nonblocking multi curl script that has a que....

__________________
.
.

mattseh

Man, I hate when people name drop their own open source scrapers.





















https://github.com/mattseh/python-web

__________________
You have a right to obey.

Berto

DO WANT. Will buzz you whenever I sober up - Thanks!

__________________
Always down to buy, sell, and trade health-related links. PM Me.
I use SerpIQ and Micro Site Masters Rank Tracker

genetic

Thanks for the PMs, had an interesting eBay scraping request that has made me rethink the way that I am doing things. Basically, the system now caches pages that are being scraped. If anybody else has any scraping requests, then please send me details and I will have a look!

@Bofu2U Sweet. I haven't started working on proxy support yet, but when I'm there I'll hit you up.

@eliquid I am using file_get_contents() for simple requests and CURL for requests where I need to send header information. I will have a look at asynchronous non-blocking requests. The system does have a queue, and can be set to process a defined number of jobs per minute.

@mattseh Thanks for sharing that, I will have a look.

@Berto Cheers for the PM, I will drop you a reply in a minute.

Bofu2U

Sounds good. You'll need about 1500-2000 for my test if you're game.

__________________

CLKeenan

Sent a PM as well

__________________
Interested in guest posting on a 9 year old domain in the education niche? PM me with what you have to offer. Looking for other guest posts, written content, design/dev work, and general SEO work.