Customizing Prometheus AlertManager notifications in Slack

Question

I have few Win servers (like Build, DB, App etc) where WMI is installed and configured, and able to read Metrics (based on rules like Disk Space >90) on my Prometheus dashboard.

Setup an Altermanager on the same box and I'm writing Metrics (Diskspace >90) details to my Slack Channel.

route:
 group_by: [cluster]
 # If an alert isn't caught by a route, send it slack.
 receiver: slack
 routes:
  # Send severity=slack alerts to slack.
  - match:
      severity: critical
    receiver: slack
receivers:
- name: slack
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/Token'
    channel: '#alerts'

And the output of Slack notification is - Actual

Is there a way where I can ready Machine Name - along with differentiating it with Tag Name - like Build, Db, etc?

I want notification to be more readable like below (I got it from some blog). How can I achieve this?

Expected

I think its good to share the alert rules... There would be labels that can give you the machine name. Alert manager just passes on the labels and the content as-is. — Neeraj Krishna, Oct 22 '18 at 04:46

score 2 · Answer 1 · answered Jan 30 '19 at 13:08

2

You can start with this nice example. Add this under slack_configs:

        text: >-
          {{ range .Alerts }}
             *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
            *Description:* {{ .Annotations.description }}
            *Graph:* <{{ .GeneratorURL }}|:chart_with_upwards_trend:> *Runbook:* <{{ .Annotations.runbook }}|:spiral_note_pad:>
            *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

answered Jan 30 '19 at 13:08

Ivan Aracki

4,861
11
59
73

Wher edid you find .GeneratorURL ? where does that go on the alert? – Justin Thomas Oct 31 '19 at 21:42
`The generatorURL field is a unique back-link which identifies the causing entity of this alert in the client.` - I've found it [here](https://prometheus.io/docs/alerting/clients/) – Ivan Aracki Oct 31 '19 at 23:05
I found that you set it with the --web.externalurl flag in Prometheus. – Justin Thomas Nov 02 '19 at 13:51

score 1 · Answer 2 · edited Jan 30 '19 at 16:12

1

Add a "text" field in the receivers section. This will allow you to pull in information from the prometheus-rules.yaml file.

receivers:
- name: slack
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/Token'
    channel: '#alerts'
    text: "<!channel> \n summary: {{ .CommonAnnotations.summary }}\n description: {{ .CommonAnnotations.description }}"

edited Jan 30 '19 at 16:12

Ivan Aracki

4,861
11
59
73

answered Oct 30 '18 at 16:36

Karen

11
4

Customizing Prometheus AlertManager notifications in Slack

2 Answers2