rotating proxies with scrapy with authentication

Question

just a noob question but I can't seem to find the answer googling. how can i use this package https://pypi.org/project/scrapy-rotating-proxies/ if the proxy requires a user/password? do I just put it in the rotating list like that?

ROTATING_PROXY_LIST = [
    'https://username:password@proxy:port',
    'https://username:password@proxy:port',
    # ...
]

thanks

[That’s correct](https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware). — Gallaecio, Feb 24 '21 at 07:10

score 2 · Answer 1 · edited Dec 30 '22 at 06:50

2

what worked for me is using the rotating proxy list path:

ROTATING_PROXY_LIST_PATH = 'path/to/proxylist.txt'

and then in the text file, I have my list formatted like this:

USERNAME:PASSWORD@HOST:PORT
USERNAME:PASSWORD@HOST:PORT
USERNAME:PASSWORD@HOST:PORT

edited Dec 30 '22 at 06:50

mozway

194,879
13
39
75

answered Dec 30 '22 at 01:02

Mubashir

21
1
3

rotating proxies with scrapy with authentication

1 Answers1