Python scraping tool

[mattseh]

New feature added https://github.com/mattseh/python-web/commit/0b2de8296551b2c9915054967a3bc7f948a23cc8

rate limit your proxies easily.

manager = ProxyManager(proxies)

then for example:

web.grab(url,manager)

it will use a proxy that hasnt been used for (by default) 20 seconds.

Great for hitting google etc

 

[Jake232]

I'm thinking I need to learn python. Got any suggestions on where to start?

I have pretty extensive knowledge in PHP, if that's going to help at all.

 

[mattseh]

The Python Tutorial — Python v2.7.2 documentation just hack at shit man

 

[j0hnsmith]

I'm thinking I need to learn python. Got any suggestions on where to start?

I have pretty extensive knowledge in PHP, if that's going to help at all.

Python is virtually pseudo code, spend an hour two following a tutorial and you should be pretty competent. The docs are great.

 

[Hammi]

The tutorial is good.

the books dive into python and learn python the hard way are pretty good.

also make stackoverflow.com and #python on freenode your new best friends.

 

[Rexibit]

I'm thinking I need to learn python. Got any suggestions on where to start?

I have pretty extensive knowledge in PHP, if that's going to help at all.

TheNewBoston – Free Educational Video Tutorials on Computer Programming and More! » Python

Follow those and you'll be golden. The tutorial on Python's website for 2.7.2 is great as well, but it's always good to watch someone doing something and hear their logic behind it.

 

[mattseh]

Latest commits:

results = multi_grab(urls)

returns all successful grabs as (url,result) tuples. Uses gevent for "threading", fast as fuck