3

I'm trying to configure into Prometheus Alerting Manager an alert that appears when the status of 2 different hosts is down. To better explain, I have these couples of hosts (host=instance):

host1.1
host1.2

host2.1
host2.2

host3.1 
host3.2

host4.1
host4.2
...

and I need an alert that appears when both hosts of the SAME couple are DOWN:

expr = ( icmpping{instance=~"hostX1"}==0 and icmpping{instance=~"hostX2"}==0 )

(I know that the syntax is not correct, I just wanted to underline that X refers to the same number on both icmppingconditions)

Any hint?

Sugaretto
  • 31
  • 2

1 Answers1

2

The easiest way is perhaps to generate a label at ingestion time reflecting this logic, using relabel_config

relabel_configs:
  - source_labels: [host]
    regex: ^(.*)\.\d+$
    target_label: host_group

It will generate the label you need for matching:

host=host1.1 => host_group=host1
host=host1.2 => host_group=host1

You can then use it for your alerting rules.

sum(icmpping) on(host_group) == 0

If this is not possible, you can use label_replace to achieve the same (only on instant vectors)

sum(label_replace(icmpping,"host_group","$1","host","(.*)\._\\d+")) on(host_group) == 0
Michael Doubez
  • 5,937
  • 25
  • 39