How to callback a Scrapy spider method upon receiving SIGINT or Scrapyd's cancel.json call?

Question

In Scrapy, when we press CTRL+C we see Received SIGINT, shutting down gracefully. Send again to force in logs that comes from this code.

Or when we request cancel.json of Scrapyd following code is executed

I want to catch those signals in my Spider, so any of those is called, one of my Spider's callback method should also be called.

I have searched a lot about this but did not find any help.

score 0 · Answer 1 · answered Sep 16 '21 at 07:44

We can use signal to catch those CTRL+C or TERM signal sent my scrapyd

import signal

def custom_terminate_spider(self, sig, frame):

    self.logger.info(self.crawler.stats.get_stats()) #print stats if you want
    
    #dangerous line, it will just kill your scrapy spider running immediately
    os.kill(os.getpid(), signal.SIGKILL)

def start_requests(self):
    signal.signal(signal.SIGINT, self.custom_terminate_spider) #CTRL+C
    signal.signal(signal.SIGTERM, self.custom_terminate_spider) #sent by scrapyd

How to callback a Scrapy spider method upon receiving SIGINT or Scrapyd's cancel.json call?

1 Answers1