1

I'm using prometheus as grafana's datasoure. I want to get growth of a my_metrics(type Count) for a given time range. For example I can calculate the increase over the last few hours:

my_metrics{label="label1"} - my_metrics{label="label1"} offset $__range

But how can I calculate the increase for given time range? For example increase for 2022/05/19 18:00:00 - 2022/05/20 00:00:00

Thanks.

Bryan Chen
  • 139
  • 1
  • 13

2 Answers2

0

Combine sum with rate. Rate will be per second, so if you sum up all rate per seconds data points over a given interval you will get the increase over a given time range:

sum by(label) (rate(my_metrics{label="label1"}[time range]))

Edit: (delta and some concrete time slot)

It seems as if the delta function is an easier way to achieve this in the case of gauges.

You will of course get a time series of computed values. To get the value for 2022/05/19 18:00:00 - 2022/05/20 00:00:00 just use an interval of 2h and get the computed value for 2022/05/20 00:00:00 by using a Table.

See answer of Lentil1016 to a similar question.

Sascha Doerdelmann
  • 648
  • 6
  • 11
  • I see...I want the dashboard to dynamically calculate the results based on the time range of grafana. So the formula should like this: `delta(my_metrics{label="label1"}[${__range}]@${__to:date:seconds})` Although the metric type is `count` but it is working well – Bryan Chen Jun 02 '22 at 03:41
  • I think I should use [`increase` function](https://prometheus.io/docs/prometheus/latest/querying/functions/#increase),there are difference between `count` and `guage`: [Link 1](https://stackoverflow.com/questions/59644317/calculate-difference-between-first-and-last-element-in-counter-metrics-range-ve) and [Link 2](https://stackoverflow.com/questions/58674087/why-there-are-both-counters-and-gauges-in-prometheus-if-gauges-can-act-as-counte/58675155#58675155) – Bryan Chen Jun 02 '22 at 03:47
  • Yes, that is an option. Using increase instead of rate is "syntactic sugar for rate(v) multiplied by the number of seconds" as the [documentation](https://prometheus.io/docs/prometheus/latest/querying/functions/#increase) says. Note that you should use sum/rate if you use function [histogram_quantile](https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile). – Sascha Doerdelmann Jun 03 '22 at 07:49
0

If you need obtaining the increase of some counter metric m over a time range (t-d .. t], then the following PromQL query can be used:

increase(m[d] @ t)

If the metric is a gauge, then just substitute increase() with delta():

delta(m[d] @ t)

These queries use @ modifier for fixing the end of the time range at t. The start of the time range - t-d - is adjusted with the d lookbehind window in square brackets.

Prometheus provides two endpoints for querying:

  • /api/v1/query aka instant query. This endpoint accepts the timestamp for the query at time argument. So the query can be simplified to just increase(m[d]) when passed to this endpoint.
  • /api/v1/query_range aka range query. This endpoint accepts start and end query args, which specify the time range for the query, alongside step query arg, which specifies the interval between returned query results. This endpoint returns independently calculated query results at the following timestamps: start, start+step, start+2*step, ... , end. If increase(m[d]) is passed to this endpoint, then this query is executed independently per each timestamp t mentioned above. Grafana queries this endpoint for building graphs. That's why every point on the graph would contain the m increase over the lookbehind window d ending at this point.

If m returns multiple time series, then both increase(m[d]) and delta(m[d]) would return individual results per each input time series. These results can be summed by wrapping the query into sum():

sum(increase(m[d]))

If you want obtaining the graph, which always starts from zero on the left side and grows with the m's increase over time on the selected time range, then Prometheus doesn't provide such a functionality. But this task can be solved with VictoriaMetrics - this is Prometheus-like monitoring system I work on:

running_sum(increase(m))

This query uses the following features of VictoriaMetrics:

  • increase() without lookbehind window in square brackets. VictoriaMetrics automatically sets the lookbehind window to the interval between points shown on the graph in this case. See these docs for more details.
  • running_sum() function for calculating the running sum over the per-point increase() results on the graph.
valyala
  • 11,669
  • 1
  • 59
  • 62