1

I am trying to implement spidermon to monitor scrapyd:

https://github.com/scrapinghub/spidermon/blob/master/spidermon/contrib/scrapy/monitors.py https://spidermon.readthedocs.io/en/latest/monitors.html#is-there-a-basic-scrapy-suite-ready-to-use

Currently it monitors minimum number of items and max amount of errors. In case of error it sends out a message via telegram. Somehow either the documentation must be wrong or my implementation as the amount of errors or items will not be included in the message:

    from spidermon import Monitor, MonitorSuite, monitors
    from spidermon.contrib.actions.telegram.notifiers import SendTelegramMessageSpiderFinished
    from spidermon.contrib.monitors.mixins import StatsMonitorMixin


    @monitors.name('Item count')
    class ItemCountMonitor(Monitor):

        @monitors.name('Minimum number of items')
        def test_minimum_number_of_items(self):
            item_extracted = getattr(
                self.data.stats, 'item_scraped_count', 0)
            minimum_threshold = 500

            msg = 'Extracted less than {} items'.format(
                minimum_threshold)
            self.assertTrue(
                item_extracted >= minimum_threshold, msg=msg
            )

    @monitors.name("Error Count Monitor")
    class ErrorCountMonitor(Monitor):
        """Check for errors in the spider log.
        You can configure the expected number of ERROR log messages using
        ``SPIDERMON_MAX_ERRORS``. The default is ``0``."""

        @monitors.name("Should not have any errors")
        def test_max_errors_in_log(self):
            errors_threshold = self.crawler.settings.getint(SPIDERMON_MAX_ERRORS, 0)
            no_of_errors = self.stats.get("log_count/ERROR", 0)
            msg = "Found {} errors in log, maximum expected is " "{}".format(
                no_of_errors, errors_threshold
            )
            self.assertTrue(no_of_errors <= errors_threshold, msg=msg)


    class SpiderCloseMonitorSuite(MonitorSuite):

        monitors = [
            ItemCountMonitor,
            # ItemValidationMonitor,
            ErrorCountMonitor
        ]

        monitors_failed_actions = [
            SendTelegramMessageSpiderFinished,
        ]

How can the msg text be included within the telegram notification?

enter image description here

Patrick Klein
  • 1,161
  • 3
  • 10
  • 23
merlin
  • 2,717
  • 3
  • 29
  • 59
  • I think `msg` only appears in the standard output, as it would be too verbose in Slack, and the documentation does not say otherwise as far as I can tell, so I don’t think there’s a documentation bug here. What you want can probably be accomplished with Spidermon templates, but I’m not familiar enough with them to help :( – Gallaecio Jun 01 '20 at 11:35
  • @Gallaecio I think there might be a thing or two missing in the documentation. The settings `SPIDERMON_SLACK_NOTIFIER_INCLUDE_OK_ATTACHMENTS` and `SPIDERMON_SLACK_NOTIFIER_INCLUDE_ERROR_ATTACHMENTS` in the [source code](https://github.com/scrapinghub/spidermon/blob/master/spidermon/contrib/actions/slack/notifiers.py) seem to fulfill a similar purpose like the ones regarding Telegram that I've mentioned in my answer. – Patrick Klein Nov 21 '20 at 23:30

1 Answers1

0

There are two options you can set in settings.py or your crawler's settings.
To include only the error messages you are asking for, you can set SPIDERMON_TELEGRAM_NOTIFIER_INCLUDE_ERROR_MESSAGES to True. If you'd also want more verbose messages in case there are no errors, you can add SPIDERMON_TELEGRAM_NOTIFIER_INCLUDE_OK_MESSAGES and set it to True as well.

I could not find any documentation for this, so I have to point you to the source code: SendTelegramMessageSpiderFinished

Adding the following to your settings.py should convince the message template to give you the full output:

SPIDERMON_TELEGRAM_NOTIFIER_INCLUDE_OK_MESSAGES = True
SPIDERMON_TELEGRAM_NOTIFIER_INCLUDE_ERROR_MESSAGES = True
Patrick Klein
  • 1,161
  • 3
  • 10
  • 23