I'm trying to setup scrapy
+ tor
I'm using scrapy 0.24.6
I first tried using
polipo
to be able to accesstor
as an http proxy (https://pkmishra.github.io/blog/2013/04/16/scrapy-run-using-tor-and-multiple-agents-part-2-ubuntu/) I'm able to configure my web browser to use polipo and I'm able to browse using TOR and I can curl. I triedHttpProxyMiddleware
and using env var or writing my own custom middleware, same result:scrapy
always returns 403Then I tried to use
tor
directly, again I can configure my web browser to use the socks proxy and I can curl withtorsocks
, butscrapy
always returns 403
Anyone has an idea about what might be wrong ?
It looks like the error come from scrapy
because I have the exact same headers/user-agent with and without tor, but through tor I always get a 403