Monitoring connectivity for Slack and PagerDuty

Question

We are using PRTG to monitor a number of internal resources, and we have set it up to alert us on a Slack channel and/or via PagerDuty (depending on severity) using their respective APIs. Considering that Slack and PagerDuty are external to us, we would also like to monitor whether our PRTG instance can access them -- basically, a form of self-monitoring or Who Watches the Watchmen?

So far the only reliable method we've found for Slack is to post an actual message to a private "testing" Slack channel, e.g. (Slack URL details redacted):

POST https://hooks.slack.com/services/XXX/YYY/ZZZ
Content-Type: application/json

{ "text": " ", "channel": "#prtg-webhook-test" }

Similarly, PagerDuty's Events API appears to be POST only, and the valid actions are limited to trigger, acknowledge, and resolve:

POST https://events.pagerduty.com/v2/enqueue
Content-Type: application/json

Is there a good way to test HTTPS connectivity without posting an actual Slack message / creating an actual PagerDuty alert? I couldn't find anything in documentation for either service, or a creative way to create an appropriate sensor in PRTG.

score 1 · Answer 1 · answered Jul 17 '18 at 23:01

1

For Slack you might rather want to make an call to the API, not to a webhook.

I would recommend using auth.test, since its one of the few methods that has no rate limit.

Also, for the whole Slack service you can see the current status on this official webpage.

answered Jul 17 '18 at 23:01

Erik Kalkoken

30,467
8
79
114

Thanks. The intent of the test is to confirm https://hooks.slack.com is accessible from the machine that should be sending notifications. If we can be certain that https://api.slack.com goes to the same place as hooks.slack.com (i.e. guarantee that if one is accessible, the other is too), and if our internal proxy device allows both (or allows wildcards i.e. *.slack.com -- something I'll need to check with a colleague), then auth.test could work for us. As for rate limiting, shouldn't be a problem for us, we're checking at most once per minute, probably every 5 minutes. – Paul Karlin Jul 18 '18 at 02:27

score 0 · Answer 2 · answered Apr 18 '22 at 21:09

For pure connectivity, you can do a POST against the Events API with an empty payload, and you'll get an error message back:

curl --location --request POST 'https://events.pd-staging.com/v2/enqueue' \
--header 'Content-Type: application/json' \
--data-raw '{}'

{
    "status": "invalid event",
    "message": "Event object is invalid",
    "errors": [
        "'event_action' is missing or blank",
        "'routing_key' must be provided in the body, or provided in the headers using 'x-routing-key'"
    ]
}

If you'd also like to validate your routing key, you can send an acknowledge event with a dummy dedup_key:

curl --location --request POST 'https://events.pd-staging.com/v2/enqueue' \
--header 'Content-Type: application/json' \
--header 'Cookie: uid=rBGA1lymclmSzRCsAwO3Ag==' \
--data-raw '{
    "routing_key": "<your_routing_key>",
    "event_action": "acknowledge",
    "dedup_key": "something_that_will_never_match_an_open_incident"
}'

{
    "status": "success",
    "message": "Event processed",
    "dedup_key": "something_that_will_never_match_an_open_incident"
}

Note that this will not show up anywhere in the PagerDuty UI, but that could be what you'd want anyways.

Monitoring connectivity for Slack and PagerDuty

2 Answers2