0

I have a hourly batch jobs in which I defined a gauge metrics where I'd set_to_current_time at certain time in the program.

However when I see it in chronosphere, it will show with gaps in the graph instead of a continuous curve.

chronosphere graph

The intent is to generate alerts once the gauge value become too old. In order to make this alert, I intend to use (time() - series) / 3600 > 1.5 as a criteria. However this won't work if series are bunch of empty values.

So the question is, is there a way to interpolate the curve to use the last non-empty value? Or is there a different way of doing it?

Previously, we were using open source prometheus pushgateway and it works correctly.

vimuth
  • 5,064
  • 33
  • 79
  • 116
Bei Zhang
  • 1
  • 1

1 Answers1

1

You can use last_over_time for your values. For example to fill the gaps in graph from screenshot you could use

last_over_time(series[3h])

It'll fill the gaps with latest value of series over last three hours.

Regarding your alert rule: its expr: would look something like

(time() - last_over_time(series[3h])) / 3600 > 1.5
markalex
  • 8,623
  • 2
  • 7
  • 32