Scrapers

[Phre0nBurn]

Ive been trying to find a scraper somewhere online, but I can't find one for the life of me...

My main goal is to hit yellowpages. I PM'd krazyjosh about his scraper script, but apparently he is on vacation, and it's been a few days...

Anyone know of anything else out there?

Obviously, the cheaper the better.

 

[capmorgan]

Web Data Extractor

I came across this one a few days ago. Now I am no expert but I hav tried a few ways to get data, including looking to have a custom script made which they wanted an arm and a leg for. This was easy to setup, and will more then likely purchase soon.

Web Extractor, Web Scraper, Web Grabber, Data Extraction, Website Ripper

 

[Enigmabomb]

Scraper: Something you pay someone else to write, and it works a little while. Or something you learn to write yourself.

 

[emp]

Yeah, learn to write it yourself.

A scraper will ALWAYS need to be updated / modified, as the concept is inherently "brittle". Basically, if your target changes, your scraper has to be changed.

::emp::

 

[xmcp123]

Easiest way is to just code a function to extract anything between argument1 and argument2, then another to get values/names of any form in the data you send to it. That way, it's rarely more than 1-2 modifications to update the scraper.

 

[flukey]

You could give Dapper a try - personally I haven't used it much but looks like it has potential:
Dapper: The Data Mapper

 

[DavidR]

I generally don't sell my software since it's basically giving weapons to competition, however, I can sell you a yellowpages database of anything you want. It's the next best thing I suppose. Not going to part with my software though.

 

[ConceptualMind]

I generally don't sell my software since it's basically giving weapons to competition, however, I can sell you a yellowpages database of anything you want. It's the next best thing I suppose. Not going to part with my software though.

You got a PM.

 

[DaveEMG]

Phreon - learn to code.

 

[Rockpile]

Has anyone used PiggyBank and Solvent to write a scraper for yellowpages?

Piggy Bank - SIMILE
Solvent - SIMILE

 

[smaxor]

I have some with simple tutorials on my site in my sig.

 

[smap]

You'll still have to write code to parse the html but this might get you somewhere ... WebReaper - Introduction.

 

[IronLogik]

excellent primer & starting point:

"Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL" by Michael Schrenk

even has his own php libs with easy scraper functions here: Official Web Site: Webbots, Spiders, and Screen Scrapers, by Michael Schrenk

 

[plepco]

I second that book...it brought me from a 90-lb PHP weakling to a screen-scraping hulk in less than 3 weeks.

But yeah: learn some PHP. Seriously, screen scrapers are one of the easiest things to write. And always useful.