Hi, being a noob here, I thought maybe it's time to contribute something.
After reading an Eli's article about thing called "Desert Scraping" with following comments below it (especially the ones about using DMOZ too), I came out with this little tool. Its purpose is simple - you enter a DMOZ listing URL (for example Open Directory - Arts: Movies: Databases), it scrapes archive.org for archived versions of that listing page and outputs websites, which were dropped from this listing in the past. Then, it checks if the sites are still indexed by Google and Yahoo to give you some idea, and also gives you a link to the archive.org results for them. You can then look into these archived sites for some good content (especially if they are not indexed anymore).
It certainly is far from being much useful or user friendly, but hell it took me 20 minutes to code, so screw it... there is probably a ton of bugs here too, but the main concept should be working. I am testing automated Copyscape checking for it right now.
Oh and one more thing - there is a 30 seconds script timeout so it's likely that the script will just die when processing larger result set.
Here is the script live: http://dev.mediaworks.cz/dmoz_dropped.php
Here is the source: http://dev.mediaworks.cz/dmoz_dropped.phps
After reading an Eli's article about thing called "Desert Scraping" with following comments below it (especially the ones about using DMOZ too), I came out with this little tool. Its purpose is simple - you enter a DMOZ listing URL (for example Open Directory - Arts: Movies: Databases), it scrapes archive.org for archived versions of that listing page and outputs websites, which were dropped from this listing in the past. Then, it checks if the sites are still indexed by Google and Yahoo to give you some idea, and also gives you a link to the archive.org results for them. You can then look into these archived sites for some good content (especially if they are not indexed anymore).
It certainly is far from being much useful or user friendly, but hell it took me 20 minutes to code, so screw it... there is probably a ton of bugs here too, but the main concept should be working. I am testing automated Copyscape checking for it right now.
Oh and one more thing - there is a 30 seconds script timeout so it's likely that the script will just die when processing larger result set.
Here is the script live: http://dev.mediaworks.cz/dmoz_dropped.php
Here is the source: http://dev.mediaworks.cz/dmoz_dropped.phps
