Problems running web scraper in Spyder IDE

Question

I have a code that is using Scrpay framework and here's the code

import scrapy
from scrapy.crawler import CrawlerProcess

class DemoSpider(scrapy.Spider):
    name = "DemoSpider"
    
    def start_requests(self):
        urls = ['http://quotes.toscrape.com/page/1/', 'http://quotes.toscrape.com/page/2/']
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)
    
    def parse(self, response):
        page = response.url.split('/')[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)
            self.log('Saved File %s' % filename)

process = CrawlerProcess()
process.crawl(DemoSpider)
process.start()

The code is working well when running like that from terminal (Windows 10 PowerShell) python demo.py. But I need to run the code using Spyder IDE. When trying I got an error like that

    ReactorBase.startRunning(self)

  File "C:\ProgramData\Anaconda3\lib\site-packages\twisted\internet\base.py", line 765, in startRunning
    raise error.ReactorNotRestartable()

ReactorNotRestartable

(*Spyder maintainer here*) Please go to the menu `Run > Configuration per file` and activate the option `Execute in an external system terminal`. That should solve your problem. — Carlos Cordoba, Nov 17 '20 at 05:31
Ok, thanks for the confirmation. I'll add it as an answer then. — Carlos Cordoba, Nov 18 '20 at 14:30

score 2 · Accepted Answer · edited Feb 09 '21 at 19:55

2

(Spyder maintainer here) Please go to the menu Run > Configuration per file and activate the option Execute in an external system terminal.

That will run your code in a regular Python interpreter, which will avoid the problems you're having to start the server that runs the scraper in our IPython console.

edited Feb 09 '21 at 19:55

DisappointedByUnaccountableMod

6,656
4
18
22

answered Nov 18 '20 at 14:50

Carlos Cordoba

33,273
10
95
124

Problems running web scraper in Spyder IDE

1 Answers1