AlertManager is not forwarding alerts to webhook receiver

Question

We have setup Prometheus in a kubernetes cluster using PrometheusOperator. We are trying to configure AlertManager using the AlertManagerConfig custom resource. We tried creating an alert route which maps to a webhook receiver and then triggering a test alert. The alert seems to be caught by AlertManager but it is not being forwarded to the webhook endpoint. AlertManager pod logs are also not printing any logs regarding notifications being send to the receivers for an alert. Sharing the test config below:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: discord-config
spec:
  receivers:
  - name: discord
    webhookConfigs:
    - url: '<webhook-url>'
      sendResolved: true
  route:
    groupBy: ['job']
    groupWait: 15s
    groupInterval: 15s
    repeatInterval: 15s
    receiver: 'discord'
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: test-rules
spec:
  groups:
  - name: test-rule-group
    rules:
    - alert: TestAlert
      expr: vector(1)
      labels: 
        severity: medium  
      annotations:
        description: "This is a reciever test for webhook alert"
        summary: "This is a dummy summary"

Is there anything else that needs to be taken care of for the receivers to start receiving alerts?

score 3 · Accepted Answer · answered May 24 '21 at 13:34

I was able to find the root cause of the issue. actually root causes. There were two problems:

I was using webhook to integrate with a Discord channel, which I later learned is not straightforward. A middle layer is required to parse and forward webhook alerts to Discord in a compatible template. A good solution is already mentioned in the Prometheus documentation, which points to alertmanager-discord application. I used the docker image for it to create a deployment and a service which bridged alertmanager to discord.
The operator was adding an additional namepsace label matcher in the top most alert route. So I added the same label to the alerts I created. I used this Routing Tree editor to visualize the routes and make sure the given set of labels matches a route.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager-discord
spec:
  selector:
    matchLabels:
      app: alertmanager-discord
  replicas: 1
  template:
    metadata:
      labels:
        app: alertmanager-discord
    spec:
      containers:
      - name: alertmanager-discord
        image: benjojo/alertmanager-discord
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 9094
        env:
          - name: DISCORD_WEBHOOK
            value: {{ .Values.webhookURL }}
---
apiVersion: v1
kind: Service
metadata:
  name: alertmanager-discord
spec:
  selector:
    app: alertmanager-discord
  ports:
  - port: 9094
    targetPort: 9094
  type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alertmanager
spec:
  receivers:
  - name: discord
    webhookConfigs:
    - url: 'http://alertmanager-discord:9094'
      sendResolved: true
.
.
.

AlertManager is not forwarding alerts to webhook receiver

1 Answers1