How to display values of metrics in alert rules prometheus

Question

I create a rule in prometheus alertmanager, which tell about least space on mountpoints in percents - and additional to this i want to show how much least space in gigabytes , but i do not want to hardcoded a mountpoint to show gigabytes, i want use $labels.mountpoint which come from a "expr" , not hardcode.

i find a similar problem on this link https://github.com/prometheus/alertmanager/issues/549 but in this case using hardcoded mountpoint

this is my rule

- alert: OutOfDiskSpace
    expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
    for: 1m
    labels:
      severity: Critical
    annotations:
      description: "Disk is almost full (< 10% left)\n {{ $labels.instance_short }}\n {{ $labels.mountpoint }}\n VALUE = {{ printf `node_filesystem_avail_bytes / 1024 / 1024 / 1024` | query | first | value | humanize }}"

when i used node_filesystem_avail_bytes / 1024 / 1024 / 1024 in VALUE, i do not take the mountpoint from expression, but i know where actual value, - it's in $labels.mountpoint which i can't use in template or don't know how to do this

score 5 · Accepted Answer · answered Aug 20 '19 at 07:20

5

  - alert: OutOfDiskSpace
      expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
      for: 5s
      labels:
        severity: Critical
      annotations:
        description: "Disk is almost full (< 10% left)\n {{ $labels.instance_short }}\n {{ $labels.mountpoint }}\n VALUE = {{ printf \"node_filesystem_avail_bytes{mountpoint='%s'}\" .Labels.mountpoint | query | first | value | humanize1024 }}"

answered Aug 20 '19 at 07:20

a1dude

174
2
13

1

If you want to exclude some mountpoint, you need to write this in section "expr" `expr: **{mountpoint!~"/mnt/videorec/.+|/pool/kvm_fast"}**` – a1dude Aug 20 '19 at 08:44

How to display values of metrics in alert rules prometheus

1 Answers1