0

I have multiple targets in prometheus which generate multiple metrics. I need to verify the values generated by a certain metric on multiple instances and trigger an alert incase the values are not equal to each other.

metric_name: treds_load_peer_db_doc_cnt

values log:

treds_load_peer_db_doc_cnt{instance="com.peer0",ip="192.168.191.2",job="prod"} 2136589 treds_load_peer_db_doc_cnt{instance="com.peer1",ip="10.121.81.38",job="prod"} 2136590 treds_load_peer_db_doc_cnt{instance="com.peer2",ip="10.121.1.57",job="prod"} 2136590

here's the query i'm using currently: treds_load_peer_db_doc_cnt{instance="com.peer0"} != ignoring(instance,ip) treds_load_peer_db_doc_cnt{instance="com.peer1"}

which works out but messes up all the labels. Is there a way to check metric in all targets at once & alert in case of miss-match?

Shonry
  • 3
  • 1

1 Answers1

1

I'd do something like:

max without(instance,ip)(treds_load_peer_db_doc_cnt) != min without(instance,ip)(treds_load_peer_db_doc_cnt)

which will generate an alert if they aren't all the same.

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
  • This totally works,Thanks Brian, Another quick question, if I had to run the same query for all instances, except ignoring one instance, in this case say instance="com.peer0 , how would I query it? – Shonry May 13 '19 at 11:35
  • Replace all `treds_load_peer_db_doc_cnt` occurrences with `treds_load_peer_db_doc_cnt{instance!="com.peer0"}`. – Alin Sînpălean May 13 '19 at 14:31
  • this works well too, Thanks. My only challenge is that when the alert is fired, IP and instance labels in the alert email notification go missing. I guess that is due to the calculations done in the query & it loses the meta-data generated. Any suggestions of how to get the instance & IP details in the email alert? – Shonry May 14 '19 at 08:23