sdanpo

Hi there,

I am currently building a web app which uses price comparison of items.
Problem is that shopzilla and others will not give access to their API unless you have enough volume in your site.

On the other hand in order to have volume in my site I need the price comparison.

How do I break this circle ?
Is there a way in ?

Are there other recommended price comparison API providers ?
(I tried google , not good enough)

Thanks

Dan

ChickenFucker

Sounds like you're stuck dealing with a base level idiot, write to someone with real power in the company. Or use other websites for data...

Kiopa_Matt

Come on, you have a brain for a reason. Be a bit creative, and create a scraper.

Go learn Python, or contract out to a Python developer.

sdanpo

Regarding scrapers ,

This is a legit site guys , and it might have potentially thousands of hits per day.
Writing a scraper which runs against shopzilla or against google or against shopping.com is possible.
Thing is it is dangerous , no?

I have written scrapers which take data which is relatively static (hit the relevant sites twice per day) , but here , it is just a big risk .

Don't you think ?

Thanks

Dan

Kiopa_Matt

It's fine that it's a legit site, but if you can't get API access, then fuck 'em, create a workaround. Scraper is fine, and not dangerous. There's a neat, new invention called proxies. Use them. Besides, these sites get millions of hits a day, and aren't going to care about the extra couple thousand you bring.

Do up a scraper, use it as a temporary measure until you get the needed traffic to be eligible for the API access.

sdanpo

You have opened my mind.
I would like to thank you .
This forums is AWESOME!!!!
:b anana_sml::mus ic06:

amateursurgeon

Ok, gonna add a disclaimer here. It's REALLY worth having a check with a lawyer before doing this. Databases qualify as copyrightable intellectual property, and you're almost certainly breaching the site's T&Cs by scraping them.

If you were just using the data for your own internal use (like analytics or research), I'd say go for it with a proxy, but because you'll be displaying their data openly on your website, and directly profiting from its use, you could be asking for trouble.

For a few hundreds bucks, it's worth being sure, because both G and Shopping.com have their own in-house counsel who need to justify their payroll by finding people to sue.

I'm not saying don't do it... I'm just saying make sure you manage your risks properly.

netbert

Today I've learned that a "Coma Prison" is an awesome anagram of "comparison."

darientroilokl

There are lots of website to make comparison with.