Scrapy doesn't load HttpProxyMiddleware at startup

Question

I have created a new project as follow

scrapy startproject test
scrapy genspider test1 example.com

and change settings.py as follow:

# -*- coding: utf-8 -*-

# Scrapy settings for test project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     http://doc.scrapy.org/en/latest/topics/settings.html
#     http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#     http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'test'

SPIDER_MODULES = ['test.spiders']
NEWSPIDER_MODULE = 'test.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'test (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'test.middlewares.MyCustomSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}

# Enable or disable extensions
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
#    'test.pipelines.SomePipeline': 300,
#}

# Enable and configure the AutoThrottle extension (disabled by default)
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

as you can see, I set

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}

but now, when I run the spider with scrapy crawl test1 I get the following output:

2017-02-15 20:37:52 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2017-02-15 20:37:52 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-02-15 20:37:52 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']

as you can see HttpProxyMiddleware is not being loaded.

Why HttpProxyMiddleware doesn't load?

Edit: I have installed scrapy 1.1.1

score 1 · Accepted Answer · answered Feb 15 '17 at 19:46

1

This middleware is enabled by default in DOWNLOADER_MIDDLEWARES_BASE

see documentation of DOWNLOADER_MIDDLEWARES_BASE

List of DOWNLOADER_MIDDLEWARES_BASE values :

{
    'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100,
    'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300,
    'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350,
    'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500,
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550,
    'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560,
    'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590,
    'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600,
    'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
    'scrapy.downloadermiddlewares.stats.DownloaderStats': 850,
    'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,
}

answered Feb 15 '17 at 19:46

Granitosaurus

20,530
5
57
82

weirdly enough even if it is working and enabled it is not displayed in the output log like your question is indicating. Maybe it's a bug with scrapy? – Granitosaurus Feb 15 '17 at 19:51
Yes, I read the docs, but it doesn't load at startup. If I remove `'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,` from `DOWNLOADER_MIDDLEWARES` the output is exactly the same, it is, HttpProxyMiddleware doesn't load – RdlP Feb 15 '17 at 19:51
@RdlP does it work though? if you add `meta={'proxy':'http://someproxy'}`. – Granitosaurus Feb 15 '17 at 19:55
I have tried to set a proxy environment variable in linux and it doesnt work (in the docs say that I can set that variable). I am using a virtual envorment for python (anaconda) – RdlP Feb 15 '17 at 20:03
@RdlP I've tried creating a new project with new spider to test this and it seems to be that the middleware is enabled by default but is not printed out at scrapy crawl. Try running https://gist.github.com/Granitosaurus/9420d2ec33b394c13a830f3a943e4b36 on a fresh project. I think it's a minor bug with scrapy. – Granitosaurus Feb 15 '17 at 20:07
You are right, HttpProxyMiddleware is enable by default but it doesn't show in the list of middleware loaded at start. Thanks – RdlP Mar 09 '17 at 19:51

score 1 · Answer 2 · answered Feb 15 '17 at 20:59

1

Have you tried setting a proxy in spider?

Do it with request object like this

req = Request(**all params here**)
req.meta['proxy'] = "proxy ip here";
yield req

answered Feb 15 '17 at 20:59

Umair Ayub

19,358
14
72
146

score 0 · Answer 3 · answered Feb 20 '17 at 01:10

Currently (as of version 1.3.2), the HttpProxyMiddleware is not enabled if neither http_proxy or https_proxy environmental variables are defined (source).

However, that doesn't mean responses can't be downloaded through proxies. The example posted by @Granitosaurus works at the downloader handler level (source).

There is a recent PR, not released to PyPI yet, which enables the middleware by default and provides a setting (HTTPPROXY_ENABLED) to disable it if necessary.

Scrapy doesn't load HttpProxyMiddleware at startup

3 Answers3