Scrapers Anonymous: I've been busy...

P

popcorn

New member
Sep 20, 2007
66
6
0
I've scraped over 6m KWs from the Google Keyword Tool (and the DB is growing at around 500k daily). So I've got all the classic stuff; CPC, search volume and so on.

I've also determined the EMD availability for each keyword that has a decent volume/CPC. Then for each KW that has at least one EMD, I've scraped the top #10 Google results, fetched SeoMoz data, scraped meta descriptions and page titles and determined a rough SEO competition score for the KW (that part is a work in progress).

Everything is driven by an SQL database, but I've made a browser based UI to view and filter the data. Voila:

Screen%20Shot%202012-06-30%20at%2015.36.36.png


I've got my own (probably obvious) uses for this data.

Curious if you folk have cool ideas for how it could be used?

Also I'm wondering if people would pay for access to such a tool, or even just a data dump.
 


People do pay for stuff like this. Check out ultimate niche finder and the products from amateursurgeon.
 
Aye, I know about Ultimate Niche Finder. I suppose my offering could be different in that you'd search an existing data set rather than crawling/fetching new data.

If I offered just the data, I could offer say 10m KWs for $300.
 
flaw3d said:
People do pay for stuff like this. Check out ultimate niche finder and the products from amateursurgeon.
Click to expand...

Thanks for the shout about amateursurgeon. Pretty amazing that they sell KWs for $17+ a pop.

starklenx said:
how do you define the value in your screenshot?
Click to expand...

It's actually nothing fancy at all. Just LMS * CPC. It's a yardstick to filter KWs by their potential economic value.

Houdas said:
Yeah, I wondered why internet was so slow lately.
Click to expand...

Yeah buddy, I was bringing down Google servers all over the place ;)

mattseh said:
hit up dchuk
Click to expand...

Shall do. Thanks!

(O_o) said:
msg the ahrefs ppl. they might be interested ?
Click to expand...

On it!

productionhead said:
ILL TAKE EM
Click to expand...

Good man. I'll PM you now.
 
This is great, could you be interested in doing this for other languages than English or do you need to know a lot of seed words? I would absolutely pay you for that favor if you could do so.
 
rusvik said:
This is great, could you be interested in doing this for other languages than English or do you need to know a lot of seed words? I would absolutely pay you for that favor if you could do so.
Click to expand...

Totally doable for sure. I need seed keywords for sure, but just something to get started with. The tool collects all the suggestions from Google's Keyword Tool and then hits all of those results one after the other. So just one seed keyword can produce infinite KWs. A larger seed list helps speed up the process by giving diversity for areas for the tool to explore. If that makes sense.
 
secretagentdad said:
Not to brag or anything, but we're sitting on mid 10s of millions daily capacity, plus deep analysis features on par with the top automated analyzers.
Click to expand...

Respect! Stumbled onto your site today and was impressed. Not sure what kind of volume you're shifting, but seems like you've got it nailed. Doubt I'll be encroaching on your turf anytime soon.

Reckon I'll be be over the 10m KW mark in the next week or so and I'm working hard on the analysis part (would be ace to compare notes!).

Ultimately I built all of this just for me and it just struck me that I should explore if there's anyone who can make use of the work also.
 
kmander said:
Respect! Stumbled onto your site today and was impressed. Not sure what kind of volume you're shifting, but seems like you've got it nailed. Doubt I'll be encroaching on your turf anytime soon.

Reckon I'll be be over the 10m KW mark in the next week or so and I'm working hard on the analysis part (would be ace to compare notes!).

Ultimately I built all of this just for me and it just struck me that I should explore if there's anyone who can make use of the work also.
Click to expand...

There are a few roads you can go down with this.

The CPC * Search Volume = Value leads to wafoo. For the love of god please don't go down it. I spend a ridiculous amount of support time reeducating people thanks to the assholes pushing that stuff.
 
secretagentdad said:
The CPC * Search Volume = Value leads to wafoo. For the love of god please don't go down it. I spend a ridiculous amount of support time reeducating people thanks to the assholes pushing that stuff.
Click to expand...

I hear you. It's a yardstick and a poor one at that. The numeric value itself is arbitrary; it's lot really that KW X is worth $50, but it can help give a relative comparison point with KW Y.

For my own business model, I needed something to help filter down the huge volume of data. I also apply a minimum/maximum search volume and CPC when I filter data to get rid of the outliers.

Understanding the CPC + search volume + chances of ranking in the top 5 + your ability to monetise is what matters most, I think.
 
You must log in or register to reply here.
Top Bottom