My intention is to add the query expressions to be displayed in descriptions or summary when alertmanager is alerting through alerting mediums. Is it possible something like this below?
- alert: OutOfDiskSpace
expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
for: 1m
labels:
severity: Critical
annotations:
description: "Disk is almost full. The expr query is {{ $labels.expr }}"
I think what you are looking for is $value which will display the value evaluated by the expression. For example consider the following rule
- alert: service_down
expr: (probe_http_status_code - 0) != 200
for: 1m
labels:
severity: 3
threshold: danger
annotations:
summary: "HTTP : {{ $value }} for URL *{{ $labels.instance }}*."
description: "*{{ $labels.instance }}* of job *{{ $labels.job }}* is down"
In the summary, you will get the status code of the API which is down.
In your case the following should work
- alert: OutOfDiskSpace
expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
for: 1m
labels:
severity: Critical
annotations:
description: "Disk is almost full. The expr query is $value"
Prometheus templating language is based on the Go templating system.
You can use this to your advantage to put Prometheus queries results in your annotations templates. For example, in an alert.rules.yml file you could use the following on the annotations.summary field:
-
alert: TestAlert
expr: go_info == 1
for: 5s
labels:
severity: warning
annotations:
summary: "go_info value: {{ printf `go_info{instance=\"%s\"}` $labels.instance | query | first | value }}"
description: "value: {{ $value }}"
On alert this will execute a promQL query for the go_info in this case in the format metric{instance=[alerted instance]}, which returns a tuple, that you can filter to the value using the processing pipes as shown above.
Source: Prometheus Documentation
