Using alert annotations in an alertmanager receiver

Question

I've got an alert configured like this:

ALERT InstanceDown
IF up == 0
FOR 30s
ANNOTATIONS {
    summary = "Server {{ $labels.Server }} is down.",
    description = "{{ $labels.Server }} ({{ $labels.job }}) is down for more than 30 seconds."
}

The slack receiver looks like this:

receivers:
- name: general_receiver
  slack_configs:
  - api_url: https://hooks.slack.com/services/token
    channel: "#telemetry"
    title: "My summary"
    text: "My description"

Is it possible to use the annotations in my receiver? This github comment indicates it is but I haven't been able to get anything to work from it.

Hi, man, can you, please, take a quick look at my [question](https://stackoverflow.com/questions/51301123/prometheus-slack-alert-with-faq-url), maybe you know how to solve it, so asking your kindly help. Thanks in advance. — d.ansimov, Jul 13 '18 at 13:08

score 13 · Accepted Answer · edited Sep 09 '16 at 17:40

You need to define your own template (I just hat to walk that path). For a brief example see: https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/

All alerts here have at least a summary and a runbook annotation. runbook contains an Wiki URL. I've defined the following template (as described in the blog article above) to include them into the slack message body:

{{ define "__slack_text" }}                                                                                                                                                
{{ range .Alerts }}{{ .Annotations.summary }}                                                                                                                                    
{{ if gt (len .Labels) (len .GroupLabels) }}({{ with .Labels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}                                             
{{ end }}<{{ (index .Alerts 0).GeneratorURL }}|Source> | {{ if .CommonAnnotations.runbook }}<{{ .CommonAnnotations.runbook }}|:notebook_with_decorative_cover: Runbook>{{ else }}<https://wiki.some.where/Runbooks|:exclamation:*NO RUNBOOK*:exclamation:>{{ end }}                                                                                
{{ end }}   
{{ define "slack.default.text" }}{{ template "__slack_text" . }}{{ end }}

The templates overwrites the slack.default.text so there is no need to reference it in the receiver configuration.

Defaults can be found in source as well as the "documentation":

For basics and details about the golang templating language:

For completeness' sake, the eventual template:

{{ define "__slack_text" }}                                                                                                                                                
{{ range .Alerts }}{{ .Annotations.text }}{{ end }}                                                                                
{{ end }}

{{ define "__slack_title" }}                                                                                                                                                
{{ range .Alerts }}{{ .Annotations.title }}{{ end }}                                                                                
{{ end }}

{{ define "slack.default.text" }}{{ template "__slack_text" . }}{{ end }}
{{ define "slack.default.title" }}{{ template "__slack_title" . }}{{ end }}

Cheers, that works perfectly. I've amended your post with the result of my specific usecase. It might make it clearer for other viewers to see how my exact input translates to the wanted output. — Jeroen Vannevel, Sep 09 '16 at 17:40

score 1 · Answer 2 · answered May 11 '20 at 12:26

I know this response is years later, but my approach was to leverage the .CommonAnnotations and labels to display the information, and the alertname to query for a runbook -

alertmanager.yml

  slack_configs:
  - api_url: 
    channel: '#XXXXXXXXXXXXXXXX'
    color: '{{ template "SLACK_MSG_COLOR" . }}'
    send_resolved: true
    title: '{{ template "SLACK_MSG_TITLE" . }}'
    text: '{{ template "SLACK_MSG_TEXT" . }}'


  pagerduty_configs:
  - routing_key: '{{ template "Global_PD_Service_Key" . }}'
    description: '{{ template "PAGERDUTY_DESCRIPTION" . }}'
    severity: '{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toLower }}{{ else }}critical{{ end }}'
    links: 
    - text: 'Prometheus'
      href: '{{ (index .Alerts 0).GeneratorURL }}'
    - text: 'Search Runbooks'
      href: '{{ template "RUNBOOK_SEARCH" . }}'

and the .tmpl

################  
# Runbook  
################ 
# Runbook Search
{{ define "RUNBOOK_SEARCH" }}https://dsmith73.github.io/101-docs/search/?q={{ .CommonLabels.alertname }}{{ end }}



################  
# Slack  
################  
# Slack Color
{{ define "SLACK_MSG_COLOR" }}{{ if eq .Status "firing" }}{{ if eq .CommonLabels.severity "critical" }}danger{{ else if eq .CommonLabels.severity "error" }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{ else }}#439FE0{{ end }}{{ else}}good{{ end }}{{ end }}


# Slack Text  
{{define "SLACK_MSG_TEXT" }}
      <!here> - {{ .CommonAnnotations.description }}
      `View:` :chart_with_upwards_trend:*<{{ (index .Alerts 0).GeneratorURL }}|Prometheus>* or :notebook:*<{{ template "RUNBOOK_SEARCH" . }}|Runbook>*

      *Details:*
      {{ range .CommonLabels.SortedPairs }}• *{{ .Name }}:* `{{ .Value }}`
    {{ end }}
{{ end }}

#Slack Summary
{{ define "SLACK_TITLE_SUMMARY" -}}
    {{- if .CommonAnnotations.summary -}}
        {{- .CommonAnnotations.summary -}}
    {{- else -}}
        {{- with index .Alerts 0 -}}
            {{- .Annotations.summary -}}
        {{- end -}}
    {{- end -}}
{{- end -}}


# Slack Title  
{{ define "SLACK_MSG_TITLE" }}
    {{ if eq .Status "resolved" }}
        {{- .Status | toUpper }} : {{ template "SLACK_TITLE_SUMMARY" . }}
    {{ else if eq .Status "firing" }}
        {{ .CommonLabels.severity | toUpper }} : {{ template "SLACK_TITLE_SUMMARY" . }}
    {{ end }}
{{ end }}

To me, this was a nice way to tie alerts to runbooks, rather than defining them in rules, or other locations...

Using alert annotations in an alertmanager receiver

2 Answers2