1

I'm trying to learn how to work with splash for scrapy. I'm doing this tutorial: https://github.com/scrapy-plugins/scrapy-splash.

I've created a scrapy project. When I run:

$ scrapy crawl spider1

everything works fine. However, when I add to my settings.py file: 

DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

I get a message saying: ModuleNotFoundError: No module named 'scrapy_splash'. I've checked whether I have scrapy_splash installed with: 

username$ pip3 show scrapy_splash
Name: scrapy-splash
Version: 0.7.1
Summary: JavaScript support for Scrapy using Splash
Home-page: https://github.com/scrapy-plugins/scrapy-splash
Author: Mikhail Korobov
Author-email: kmike84@gmail.com
License: BSD
Location: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
Requires:

I've tried to import scrapy_splash into my spider script and into my settings script. If I do so, I get a message saying:

raise KeyError("Spider not found: {}".format(spider_name))
KeyError: 'Spider not found: spider1'

Does anyone know how to fix this issue?

titusAdam
  • 779
  • 1
  • 16
  • 35
  • Are you sure you are runing python3 scrapy? could you do `pip3 show scrapy`? It could be that you have scrapy-splash on your python3 and scrpay on python2. Also I'd highly recommend using virtualenvironment to avoid such issues :) – Granitosaurus Jan 11 '17 at 20:14
  • @Granitosaurus Hi! Yes I have scrapy 1.3 installed. I think I tried it with a virtual environment already! Will do it again. Thanks – titusAdam Jan 11 '17 at 20:43

1 Answers1

3

You should add to requirements.txt:

scrapy-splash==0.7.2

After that build the container

docker-compose build container_name

That works for me.

Steferson Ferreira
  • 343
  • 2
  • 7