1

I'm having a problem. When I set up alert rules in prometheus, I had a disk space monitoring rule that showed me the percentage of remaining space and humanize the value of the node_filesystem_avail_bytes metric.

Now my rule looks like this:

   - alert: HostOutOfDiskSpace_test
     expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes > 0 and ON (instance, device, mountpoint) node_filesystem_readonly == 0 and {instance=~"10.22.22.18.*"}
     for: 1s
     labels:
       severity: high
     annotations:
       summary: "Host {{ $labels.instance }} out of memory \n - Device: {{ $labels.device }} \n - Mountpoint - {{ $labels.mountpoint }}"
       description: "Node memory is filling up. Value = {{ $value | printf `%.2f` }} ({{ printf \"node_filesystem_avail_bytes{mountpoint='%s', instance='%s'}\" .Labels.mountpoint .Labels.instance | query | first | value | humanize1024 }})"

Here are examples of notifications from telegrams:

 [PROBLEM] HostOutOfDiskSpace_test_1
Severity: high
Summary: Host 10.11.11.100:9100 out of memory
    - Device: /dev/sda1
    - Mountpoint - /boot
Description: Node memory is filling up. Value = 72.62 (736.4Mi)
Starts at: 2023-08-31 13:19:52.25 +0300 EAT

When I try to transfer this description to grafana, it outputs it as text.

 [PROBLEM] Test_grafana_rules_filesystem
Severity: high
Summary: Host 10.11.11.100:9100 out of memory
    - Device: /dev/sda1
    - Mountpoint - /boot
Description: Node memory is filling up. Value = 72.61% ({{ printf \"node_filesystem_avail_bytes{mountpoint='%s', instance='%s'}\" .Labels.mountpoint .Labels.instance | query | first | value | humanize1024 }}) 
Starts at: 2023-08-31 13:22:49.12 +0300 EAT

enter image description here enter image description here

My question is this. How can I display the actual values of the value of the node_filesystem_avail_bytes metric in description grafana?

markalex
  • 8,623
  • 2
  • 7
  • 32
Stanislav_iz
  • 21
  • 1
  • I doubt Grafana supports `query` operation, or at least that it is that easy to use. But here you probably don't need to use it: you can extract separate metrics into separate queries, and incorporate arithmetics through Grafana's expressions. That way you'll be able to use results of any step in your alert without additional quering on summary creation step. – markalex Aug 31 '23 at 13:21

0 Answers0