Prometheus Absent function

Question

I want to check if a certain metric is not available in prometheus for 5 minute.

I am using absent(K_KA_GCPP) and giving a 5 minute threshold. But it seems I cannot group the absent function on certain labels like Site Id.

Absent works if the metric is not available for all 4 site Ids. I want to find out if the metric is not available or absent for 1 site id out of all 4 and I don't want to hardcode the site Id labels in the query, it should be generic. Is there any way I can do that?

I'm doing the same, the only other way I found to do absent on up was to do an alert on up{metric} == 0 with a time a 30secondes or less to launch the alert on first call — night-gold, Nov 29 '18 at 15:45

score 4 · Answer 1 · answered Oct 08 '20 at 16:12

I was able to achive this by doing somthing like this:

count(up{job="prometheus"} offset 1h) by (project) unless count(up{job="prometheus"} ) by (project)

If the metric is missing in the last 1 hour, it will trigger an alert. You can add any labels you need after the by section (that's helpful in altering for example).

Source: Prometheus Alert for missing metrics and labels

score 2 · Answer 2 · answered Mar 28 '22 at 09:05

2

The offset I feel like is a great starting point, but it has a big weakness. If there's no sample in the time - offset then your query doesn't return what you'd like to.

I reworked the answer from Ahmed to this:

group(present_over_time(myMetric{label1="asd"}[3h])) by (labels) unless group(myMetric{label1="asd"}) by (labels)

using period with present_over_time should fix that aforementioned problem
group() aggregation, since you don't need the value
also I like to use the actual metric, since up{} is a state of the scraped target, not the "metric is present" information which I feel might not be equivalent

answered Mar 28 '22 at 09:05

strudelPi

51
5

+1 @strudelPi. The only thing I modified - I extended 3h window (I made it 48h for my alert), cause this window defines how long your expression will be detecting the problem. If metric is still missing for a period of time more than this window length, expression will stop producing the outcome you rely on. `(group(present_over_time(st_Worker_Queue_Size_Max{deployment="worker"}[48h])) by (deployment, namespace)) unless group(st_Worker_Queue_Size_Max{deployment="worker"}) by (deployment, namespace)` – Ilya May 03 '23 at 01:12

score 1 · Answer 3 · answered Oct 04 '22 at 13:35

1

There exists the Prometheus absent_over_time function

answered Oct 04 '22 at 13:35

Gabriele Iannetti

119
1
3

score 0 · Answer 4 · answered Dec 03 '22 at 01:18

You can use it as a group! see how to configure an alert rule group

You can also use absent_over_time function

absent returns just one result as it is for a single site ID in your case

 absent(<expr>)

Returns an empty vector if the vector passed to it has any elements and a 1-element vector with the value 1 if the vector passed to it has no elements. This is useful for alerting on when no time series exist for a given metric name and label combination.

Prometheus Absent function

4 Answers4