Use conditional operator in Prometheus alert rules to set severity

Question

I would like to use the conditional operator into the Prometheus alert.rules definition to set a specific severity. For instance, if the environment is production, I want to set the severity to critical else another value.

Something like:

- alert: CPU load
  expr: expression_used_to_check_load
  for: 15m
  labels:
     severity: if $labels.env == prd -> severity = critical else something else
  annotations:
     summary: Just a summary
     description: "Just a description"

score 0 · Answer 1 · edited Nov 02 '21 at 15:39

0

You could divide the rule in two:

- alert: CPU load
  expr: expression_used_to_check_load and {env=="prd"}
  for: 15m
  labels:
    severity: critical
  annotations:
    summary: Just a summary
    description: "Just a description"
- alert: CPU load
  expr: expression_used_to_check_load and {env!="prd"}
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: Just a summary
    description: "Just a description"

edited Nov 02 '21 at 15:39

Stephen Ostermiller

23,933
14
88
109

answered Oct 04 '21 at 11:56

Marcelo Ávila de Oliveira

19,950
3
39
50

2

Is duplicating that much information really the only way to do this? – Stephen Ostermiller Nov 02 '21 at 09:57

score 0 · Answer 2 · answered Nov 02 '21 at 15:37

0

Rather than configuring this in the severity itself, you can configure the receiver of your pages differently based on environment.

- match: {owner: 'MYTEAM'}
    receiver: 'blackhole'
    routes:
    - match: {severity: 'critical', env: 'staging'}
      receiver: 'myteam-warn'
    - match: {severity: 'critical', env: 'prd'}}
      receiver: 'myteam-page'

The different receivers can be configured differently. myteam-warn might just send you an email whereas myteam-page might trigger a page.

answered Nov 02 '21 at 15:37

Stephen Ostermiller

23,933
14
88
109

I didn't understand, he wants to set "severity = warning" if "env != prd", how your solution does that? – Marcelo Ávila de Oliveira Nov 02 '21 at 20:50
The severity usually means the different paging rules are used. Rather than changing the severity you can configure different paging rules for the same severity based on staging versus production. – Stephen Ostermiller Nov 02 '21 at 22:25

score 0 · Answer 3 · answered Jun 08 '23 at 09:42

You can customize the severity this way.

- alert: KubernetesStatefulsetReplicasMismatch
    expr: 'kube_statefulset_status_replicas_ready != kube_statefulset_status_replicas'
    for: 3s
    labels:
      severity: "{{ if eq .Labels.cluster \"prod\" }}critical{{ else if eq .Labels.cluster \"dev\" }}warning{{ end }}"
    annotations:
      summary: Kubernetes StatefulSet replicas mismatch (cluster {{ $labels.cluster }})
      description: "A StatefulSet does not match the expected number of replicas.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

Use conditional operator in Prometheus alert rules to set severity

3 Answers3