0

I have a portia scrapy project at ~/portia/slyd/data/projects/scrap setup to use scrap.middlewares.RandomUserAgentMiddleware in DOWNLOADER_MIDDLEWARES, RandomUserAgentMiddleware is defined in ~/portia/slyd/data/projects/scrap/middlewares.py.

After deploying the project, the resulting egg doesn't contain the middlewares.py, so I get this error:

ImportError: Error loading object 'scrap.middlewares.RandomUserAgentMiddleware': No module named scrap.middlewares

How do I tell scrapyd-deploy to take the custom middlewares as well?

Edit:

My settings.py is this: http://pastebin.com/8gFU3YEw

I also tried manually including the middlewares.py in the generated egg, but it doesn't solve it, still the same error.

localhost
  • 55
  • 1
  • 6

1 Answers1

0

They should be placed in settings.py file. The DOWNLOADER_MIDDLEWARES variable should be declared containing a list of custom middleware to be used. Here an example that should cover your case:

DOWNLOADER_MIDDLEWARES = {
    'scrap.middlewares.RandomUserAgentMiddleware': 5,
}

More info can be found here in the official scrapy documentation.

Bernardo Botella
  • 193
  • 1
  • 8