Prometheus alertmanager multiply notifications mattermost

Question

I'm using prometheus with alertmenager and have confusing problem. Notifications one is my custom and another with [Firing] title.

Below example and configuration. I would like to have only my custom alert.

2:47 PM
[FIRING:1] (InstanceDown labels critical)
2:47 PM
custom notification

alert.rules

groups:
- name: targets
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 30s
    labels:
      severity: critical
      summary: {{ labels.instance }}

alertmanager.yml

global:
  slack_api_url: xyz

route:
  group_interval: 1m
  repeat_interval: 5m 
  receiver: 'backend'
  group_by: ['alertname', 'instance', 'application']
  routes:
    - match:
        severity: critical
        receiver: 'backend'

receivers:
  - name: 'backend'
    slack_configs:
    - send_resolved: true
    - title: '{{ .CommonAnnotations.summary }}'

score 0 · Answer 1 · answered Aug 31 '18 at 08:06

Whether or not given alert rule is firing (evaluated to meet the requirements to trigger an alert) largely depend on your search term, in your case this is 'up == 0'. This will basically trigger alert if any target (discovered as part of Prometheus service discovery or defined as static target) match the condition of the search term.

Probably you have more than one target that for one reason or another are evaluated as not being up.

Prometheus alertmanager multiply notifications mattermost

1 Answers1