0

I'm already using urllib2 to get the pages using proxy, but it's taking way too long, and I know that using proxy takes longer, but still is taking too long comparing if I test the proxy in firefox or ie.

Thanks.

Barbara
  • 1
  • 1
  • Modern web-browsers use HTTP Pipelining in which they send out multiple requests to ensure quicker load times. You could also try using multiple threads to perform the scraping in parallel. I dont see any relation to the proxy though. – Anirudh Ramanathan Jun 24 '12 at 00:44

1 Answers1

1

To download pages faster you need to do that concurrently no matter using proxy or not. Try gevent library. There is an example for concurent download https://bitbucket.org/denis/gevent/src/tip/examples/concurrent_download.py

Dmitry Trofimov
  • 7,371
  • 1
  • 30
  • 34