Prometheus operator AlertmanagerConfig unknown field "webhook_configs"

Question

After applying these manifests https://github.com/prometheus-operator/kube-prometheus/blob/main/kustomization.yaml I want to create AlertManager webhook:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: custom
spec:
  route:
    receiver: custom
    groupBy: ['job']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
  receivers:
    - name: custom
      webhook_configs:
        - send_resolved: true
          url: https://example.com

getting an error:

error: error validating "alertmanagerconfig.yaml": error validating data: ValidationError(AlertmanagerConfig.spec.receivers[0]): unknown field "webhook_configs" in com.coreos.monitoring.v1alpha1.AlertmanagerConfig.spec.receivers; if you choose to ignore these errors, turn validation off with --validate=false

How to fix that?

score 2 · Answer 1 · answered May 23 '21 at 17:43

The problem is that you are using the configuration reference for the actual AlertManager application and not the one for the Kubernetes Custom Resource provided by the Prometheus Operator.

The Prometheus Operator takes the configuration provided in the form of custom resources like AlertManagerConfig and converts them into actual AlertManager config and updates the configuration file the application uses. That is a part of the reason why you use an operator in the first place. It makes these things convenient for you.

So the actual configuration reference you should be using here is this. This part of the Prometheus Operator's github readme lists down the available custom resources you can use with it.

score 1 · Answer 2 · answered Mar 01 '21 at 12:34

Taking this link as reference,

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: custom
spec:
  receivers:
    - name: custom
      webhookConfigs:
        - send_resolved: true
          url: https://example.com

You can also check field using kubectl explain command

kubectl explain alertmanagerconfig.spec.receivers

score 1 · Answer 3 · answered Nov 13 '21 at 19:50

In addition to the answer from Kiran (Thanks!) one small correction - it is actually sendResolved and not send_resolved.

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: custom
spec:
  receivers:
    - name: custom
      webhookConfigs:
        - sendResolved: true
          url: https://example.com

score 0 · Accepted Answer · answered Nov 15 '21 at 09:01

I solved this issue by generating Kubernetes secret named alertmanager-main using this yaml:

global:
  #skipped
route:
  group_by: ['job']
  receiver: Default
  routes:
    - receiver: Watchdog
      repeat_interval: 1m
      match:
        alertname: Watchdog  

receivers:
  - name: Watchdog
    webhook_configs:
      - url: https://my.watchdog.webhook.url

  # Other things

This is Alert Manager config.

I am answering my question after some time and probably this issue with the Prometheus Operator CRD is fixed by now.

Prometheus operator AlertmanagerConfig unknown field "webhook_configs"

4 Answers4