3

We have a multi tenant Kubernetes cluster and using Prometheus Alertmanager to send alerts to those tenants via slack.

So we have config that includes this:

slack_configs:
 - send_resolved: true
   channel: '{{ printf "topic-svc-%.11s" (index (index .Alerts 0).Labels "namespace") }}'

(The %.11s ensures the channel name stays within the 21 character limit)

This works great if the slack channel exists, but if the channel doesn't exist the alert ends up in the ether (not good for an alert!).

Alertmanager logs are pretty limited in what they tell you, e.g. it's a generic error with no user data:

alertmanager-k8s-0 alertmanager level=error ts=2018-11-09T15:01:52.134984182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="cancelling notify retry for \"slack\" due to unrecoverable error: unexpected status code 404"

Tried all sorts of options, checked StackOverflow but seems all examples have simple fixed name for slack channels

Steve Goodenough
  • 161
  • 1
  • 7
  • Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that. – Steve Goodenough Nov 09 '18 at 15:16
  • This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find any `channel_not_found` errors from Slack Incoming Webhook calls. – Adil B Nov 09 '18 at 18:19
  • Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them. – Steve Goodenough Nov 12 '18 at 09:28
  • Try out this tool: https://prometheus.io/webtools/alerting/routing-tree-editor/ Maybe it will help to find a routing problem. – mibrl12 Nov 12 '18 at 12:36
  • Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is. – Steve Goodenough Nov 13 '18 at 15:22
  • Hey! @SteveGoodenough did you find an answer to your problem? I am getting a similar use-case, where I am setting up 1 channel per application, and would like to use templating based on the labels.app name for it :) – François Nov 22 '22 at 22:10

0 Answers0