How to monitor specific spiders with Spidermon?

Question

I'm following Spidermon's documentation on monitoring Spiders, in their examples, the monitors that they create seem to run across all of their Spiders, I can't figure out how to run a monitor on a single Spider.

I've tried passing my Spider into test_min_items, but I don't know where to set this.

from spidermon import Monitor, MonitorSuite, monitors

@monitors.name('Item count')
class ItemCountMonitor(Monitor):

    @monitors.name('Minimum number of items')
    def test_min_items(self):
        item_extracted = getattr(
            self.data.stats, 'item_scraped_count', 0)
        minimum_threshold = 10

        msg = 'Extracted less than {} items'.format(
            minimum_threshold)
        self.assertTrue(
            item_extracted > minimum_threshold, msg=msg
        )


class SpiderCloseMonitorSuite(MonitorSuite):

    monitors = [
        ItemCountMonitor
    ]

score 6 · Answer 1 · answered Jul 09 '19 at 11:34

There different ways to achieve this.

Every monitor test method has a data.spider property (https://spidermon.readthedocs.io/en/latest/monitors.html#id1) that contains the instance of the spider you are running. You could include an if-statement in your monitor verifying its name and doing what you want according to your monitor:

@monitors.name('Item count')
class ItemCountMonitor(Monitor):

    @monitors.name('Minimum number of items')
    def test_min_items(self):
        if self.data.spider.name == "somespider":
            ...  # Do something
        else:
            ...  # Do other thing

You can have different monitors enabled for different spiders using custom-settings (https://docs.scrapy.org/en/latest/topics/settings.html#settings-per-spider).

# spider1.py
class MySpider1(scrapy.Spider):
    name = 'myspider1'

    custom_settings = { 
        'SPIDERMON_SPIDER_CLOSE_MONITORS': (
            'myproject.Spider1MonitorSuite',
        )   
    }   

# spider2.py
class MySpider2(scrapy.Spider):
    name = 'myspider2'

    custom_settings = { 
        'SPIDERMON_SPIDER_CLOSE_MONITORS': (
            'myproject.Spider2MonitorSuite',
        )   
    }

And for each MonitorSuite you include only the monitors that makes sense for wach spider.

Specifically to a monitor that verifies a minimum number of items, we have a built-in monitor (https://spidermon.readthedocs.io/en/latest/monitors.html#spidermon.contrib.scrapy.monitors.ItemCountMonitor) that you can use. So you can set different SPIDERMON_MIN_ITEMS setting for each spider using the custom_setting mentioned previously.

How to monitor specific spiders with Spidermon?

1 Answers1