4

I'm trying to setup scrapy + tor

I'm using scrapy 0.24.6

  1. I first tried using polipo to be able to access tor as an http proxy (https://pkmishra.github.io/blog/2013/04/16/scrapy-run-using-tor-and-multiple-agents-part-2-ubuntu/) I'm able to configure my web browser to use polipo and I'm able to browse using TOR and I can curl. I tried HttpProxyMiddleware and using env var or writing my own custom middleware, same result: scrapy always returns 403

  2. Then I tried to use tor directly, again I can configure my web browser to use the socks proxy and I can curl with torsocks, but scrapy always returns 403

Anyone has an idea about what might be wrong ?

It looks like the error come from scrapy because I have the exact same headers/user-agent with and without tor, but through tor I always get a 403

eLRuLL
  • 18,488
  • 9
  • 73
  • 99
vieux
  • 23,887
  • 3
  • 26
  • 26

0 Answers0