Alertmanager failing due to "unmarshal errors: line 19: field matchers not found in type config.plain"

Question

I am trying to add custom alert-routing config to my alertmanager, deployed as a part of kube-prometheus-stack. But prometheus-operator pod, while trying to generate the alertmanager configmap, fails due to the following error:

level=error ts=2021-05-31T06:29:38.883470881Z caller=klog.go:96 component=k8s_client_runtime func=ErrorDepth msg="Sync \"infra-services/prometheus-operator-kube-p-alertmanager\" failed: provision alertmanager configuration: base config from Secret could not be parsed: yaml: unmarshal errors:\n line 19: field matchers not found in type config.plain"

I also validated the same using amtool inside alertmanager container, which gives the same error. Here is my alertmanager.yml file:

global:
  resolve_timeout: 5m
  slack_api_url: https://hooks.slack.com/services/xxxxxx/yyyyy/zzzzzzzzzzz
receivers:
- name: slack-notifications
  slack_configs:
  - channel: '#alerts'
    send_resolved: true
    text: '{{ template "slack.myorg.text" . }}'
- name: blackhole-receiver
route:
  group_by:
  - alertname
  group_interval: 5m
  group_wait: 30s
  receiver: blackhole-receiver
  repeat_interval: 12h
  routes:
  - matchers:
    - severity=~"warning|critical"
    receiver: slack-notifications
templates:
- /etc/alertmanager/config/*.tmpl

I have followed https://prometheus.io/docs/alerting/latest/configuration/ and https://github.com/prometheus/alertmanager/blob/master/doc/examples/simple.yml to write my simple alertmanager config.

score 10 · Accepted Answer · answered May 31 '21 at 14:58

10

Try to change from:

  routes:
  - matchers:
    - severity=~"warning|critical"
    receiver: slack-notifications

To:

  routes:
    - match_re:
        severity: "warning|critical"
      receiver: slack-notifications

answered May 31 '21 at 14:58

Marcelo Ávila de Oliveira

19,950
3
39
50

This definitely solved the issue for me instantly. But I found that "match_re" is deprecated as per https://prometheus.io/docs/alerting/latest/configuration/#route . Is it safe to use it, as future versions of alertmanager might remove the field? Also, any idea why is "matchers" is not allowed, or what's wrong with the previous configuration? – Tapan Halani Jun 01 '21 at 05:01

score 3 · Answer 2 · answered Aug 02 '21 at 02:37

3

upgrade to (at least) quay.io/prometheus/alertmanager:v0.22.2 and it'll definitely work.

you shouldn't continue using match_re since it's deprecated.

answered Aug 02 '21 at 02:37

timfeirg

1,426
18
37

Alertmanager failing due to "unmarshal errors: line 19: field matchers not found in type config.plain"

2 Answers2