3

I am having this error when I run a crawl process multiples times. I am using scrapy 2.6 This is my code:

from scrapy.crawler import CrawlerProcess
from football.spiders.laliga import LaligaSpider
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(settings=get_project_settings())
for i in range(1, 29):
    process.crawl(LaligaSpider, **{'week': i})
process.start()
pppery
  • 3,731
  • 22
  • 33
  • 46
Leandro Hernández Mira
  • 768
  • 9
  • 23

4 Answers4

7

For me this worked, I put it before the CrawlerProcess

import sys    
if "twisted.internet.reactor" in sys.modules:
    del sys.modules["twisted.internet.reactor"]
Mohammad-Reza Malekpour
  • 787
  • 10
  • 18
Mersad
  • 71
  • 2
4

This solution avoids use of CrawlerProcess as stated in the docs. https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script

There’s another Scrapy utility that provides more control over the crawling process: scrapy.crawler.CrawlerRunner. This class is a thin wrapper that encapsulates some simple helpers to run multiple crawlers, but it won’t start or interfere with existing reactors in any way.

It’s recommended you use CrawlerRunner instead of CrawlerProcess if your application is already using Twisted and you want to run Scrapy in the same reactor.

from twisted.internet import reactor
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
from scrapy.utils.log import configure_logging

from football.spiders.laliga import LaligaSpider

# Enable logging for CrawlerRunner
configure_logging()

runner = CrawlerRunner(settings=get_project_settings())
for i in range(1, 29):
    runner.crawl(LaligaSpider, **{'week': i})

deferred = runner.join()
deferred.addBoth(lambda _: reactor.stop())

reactor.run()  # the script will block here until all crawling jobs are finished
Md. Mujtaba Raza
  • 61
  • 2
1

I have encountered this problem, but it is solved after updating both Scrapy and Twisted. the current version of the packages. Twisted==22.8.0 Scrapy==2.6.2

Ahmed Ellban
  • 156
  • 6
0

I've just run into this issue as well. It appears that the docs at https://docs.scrapy.org/en/latest/topics/practices.html are incorrect in stating that CrawlerProcess can be used to run multiple crawlers built with spiders, since each new crawler attempts to load a new reactor instance if you give it a spider. I was able to get my code to work by using CrawlerRunner instead, as also detailed on the same page.

import scrapy
from twisted.internet import reactor
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings

class MySpider1(scrapy.Spider):
    # Your first spider definition
    ...

class MySpider2(scrapy.Spider):
    # Your second spider definition
    ...

configure_logging()
settings = get_project_settings() # settings not required if running
runner = CrawlerRunner(settings)  # from script, defaults provided
runner.crawl(MySpider1) # your loop would go here
runner.crawl(MySpider2)
d = runner.join()
d.addBoth(lambda _: reactor.stop())
reactor.run() # the script will block here until all crawling jobs are finished
z3r0fox
  • 9
  • 1