0

I deploy scrapyd in KubeSphere, I got expectation when I run scrapy and selenium:

2022-03-16T12:57:15+0000 [Launcher,1832/stderr]     return Crawler(spidercls, self.settings, init_reactor=True)
          File "/usr/local/lib/python3.9/site-packages/scrapy/crawler.py", line 82, in __init__
            default.install()
          File "/usr/local/lib/python3.9/site-packages/twisted/internet/epollreactor.py", line 256, in install
2022-03-16T12:57:15+0000 [Launcher,1832/stderr]     installReactor(p)
          File "/usr/local/lib/python3.9/site-packages/twisted/internet/main.py", line 32, in installReactor
2022-03-16T12:57:15+0000 [Launcher,1832/stderr]     raise error.ReactorAlreadyInstalledError("reactor already installed")
        twisted.internet.error.ReactorAlreadyInstalledError: reactor already installed

I don't have twisted installed separately. Why does it report that it already exists?

Dmitriy Zub
  • 1,398
  • 8
  • 35
何元斌
  • 1
  • 1

1 Answers1

0

Without code and package version, it is hard to guess what happens. If your program was once working, check your scrapy version. There is an issue in latest 2.6 version. You could pin your scrapy at 2.5.1 .

If you are under developing scrapy scripts, you might want to run multiple spiders in a script. It is easy to mess up the pipeline and get twisted.internet.error.ReactorAlreadyInstalledError error. You could check this.

Nothing
  • 13
  • 5