0

I am using ProxyMiddleware in scrapy , and its throwing non traceable error. Here is Traceback :

Unhandled error in Deferred:   
[twisted] CRITICAL: Unhandled error in Deferred:
[twisted] CRITICAL:
Traceback (most recent call last):

File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "/usr/local/lib/python3.5/dist-packages/scrapy/crawler.py", line 80, in crawl   
self.engine = self._create_engine()
File "/usr/local/lib/python3.5/dist-packages/scrapy/crawler.py", line 105, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "/usr/local/lib/python3.5/dist-packages/scrapy/core/engine.py", line 69, in init
self.downloader = downloader_cls(crawler)
File "/usr/local/lib/python3.5/dist-packages/scrapy/core/downloader/init.py", line 88, in init    
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "/usr/local/lib/python3.5/dist-packages/scrapy/middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)    
File "/usr/local/lib/python3.5/dist-packages/scrapy/middleware.py", line 40, in from_settings
mw = mwcls()    
TypeError: init() missing 1 required positional argument: 'arg'

Here are files :

settings.py

HTTP_PROXY = 'http://127.0.0.1:8123'

DOWNLOADER_MIDDLEWARES = {
#Tor Middleware
'RaidForums.middlewares.ProxyMiddleware': 400
 }

 SPIDER_MIDDLEWARES = {
'RaidForums.middlewares.ProxyMiddleware': 400
 }

middelwares.py

class ProxyMiddleware(object):
      def process_request(self, request, spider):
          request.meta['proxy'] = settings['HTTP_PROXY']
      def __init__(self, arg):
          super(ProxyMiddleware, self).__init__()
          self.arg = arg
8-Bit Borges
  • 9,643
  • 29
  • 101
  • 198
Vikram Nimbalkar
  • 11
  • 1
  • 1
  • 3

1 Answers1

0

In your settings.py, enable scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware with a higher priority value than your custom (401 or greater), so it's further from the engine than your custom middleware.

In your middlewares.py, you can access the settings with spider.settings[], just settings[] won't work. I also think you can take the __init__() out of your ProxyMiddleware altogether.

See this example from scrapinghub to get an idea of how close you are.

pwinz
  • 303
  • 2
  • 14