31

Prometheus is built around returning a time series representation of metrics. In many cases, however, I only care about what the state of a metric is right now, and I'm having a hard time figuring out a reliable way to get the "most recent" value of a metric.

Since right now it's getting metrics every 30 seconds, I tried something like this:

my_metric[30s]

But this feels fragile. If metrics are dated any more or less than 30 seconds between data points, then I either get back more than one or zero results.

How can I get the most recent value of a metric?

Cory Klein
  • 51,188
  • 43
  • 183
  • 243
  • 2
    If you need to obtain the most recent metric value outside the last 5 minutes time range, then try something like `last_over_time(metric[1h])` from [MetricsQL](https://victoriametrics.github.io/MetricsQL.html). This query increases lookback window from 5 minutes to a hour (see `1h` in square brackets). – valyala Dec 05 '20 at 22:33

7 Answers7

22

All you need is my_metric, which will by default return the most recent value no more than 5 minutes old.

030
  • 10,842
  • 12
  • 78
  • 123
brian-brazil
  • 31,678
  • 6
  • 93
  • 86
  • 13
    The Prometheus console does only show the latest value, but when querying the metric via the API (through Grafana for example), `my_metric` returns the time series. Try hitting `/api/v1/query?query=my_metric`. It returns the time series, not the most recent value. – Cory Klein Nov 21 '16 at 21:54
  • 1
    No it doesn't, http://demo.robustperception.io:9090/api/v1/query?query=up for example is only the most recent value. – brian-brazil Nov 22 '16 at 08:15
  • I'm not getting any response from your link. But here's a screenshot of my query returning a vector. http://imgur.com/OkaV0Ea – Cory Klein Nov 22 '16 at 15:56
  • 1
    That result is as expected, each is the most recent result for that time series. – brian-brazil Nov 22 '16 at 19:14
  • 3
    I'm still getting more than the latest... on prometheus i see three values returned (one for each of my apps), on grafana i get a table every scrape for last 5min – Riley Guerin Oct 18 '17 at 17:01
  • Is it possible to get the value of the last datapoint in the time-series for an alert. The $value variable gives the expression value, not the time-series value. I am trying something like the following in my alert ruies: ```description: "service {{ $labels.service }} in {{ $labels.env}} environment failed at {{$metric_name}} ```" – barakbd Jun 25 '19 at 00:02
  • 6
    @brian-brazil what if I want get a gauge's most recent value while the target is down for sometime. – WestFarmer Jul 10 '19 at 13:17
17

If you're using Prometheus directly using the query_range API endpoint you will get time series. If you switch to the query API endpoint you will get the last value.

In Grafana you can switch from time series to last value by switching the Instant-toggle.

thoredge
  • 12,237
  • 1
  • 40
  • 55
  • 3
    thanks for the note on grafana "instant" toggle – boyvinall Sep 21 '22 at 10:29
  • 1
    This is the right answer if you want to consider only the latest value, regardless of the period of time selected by the user. In our case it's necessary to show the state of the service on a Stat panel, and we don't want users to get alarmed by the red colour selecting `Last 7 days`, which will make Grafana consider alerts since 7 days ago, though they are already resolved. Thanks @thoredge – Sergio Pelin Jul 13 '23 at 11:07
6

Given this:

namespace_metricname_count_sum{id="1",status="to-do"}
namespace_metricname_count_sum{id="1",status="in-progress"}

If you want to get the most recent value you need to use the value that has in common in this case is id=~".*" by grouping the logs you will be able to get the last value in a time range 

count ( max_over_time ( namespace_metricname_count_sum{id=~".*"}[12h])) by (status)
Axel Monroy
  • 359
  • 4
  • 3
6

I had a similar issue with metrics I was getting from AWS via prom/cloudwatch-exporter. It seems AWS takes awhile to converge its CloudWatch metrics. It used to be about 10 minutes but now it's more like 13 minutes. We've been missing issues like disk space low because these metrics utterly fail to make it to prometheus, therefore our alerts were useless.

I found "offset" useful here, where I wanted the last metric but it was outside of the 5m cutoff. So by specifying an offset, I can still pick up a value instead of nothing. Example:

aws_ec2_cpuutilization_average offset 15m
Eric McCormick
  • 331
  • 3
  • 4
5

By default Prometheus returns the most recent values if they were posted during the last 5 minutes. This lookbehind window is known as lookback delta and it can be configured with --query.lookback-delta command-line flag. See these docs for more information. The lookbehind window can be augmented on a per-query basis via last_over_time() function. For example, the following query returns the most recent values for time series with my_metric name if these values were available during the last hour:

last_over_time(my_metric[1h])
valyala
  • 11,669
  • 1
  • 59
  • 62
0

To get the most recent value of my_metric older than 5m without resorting to hacky PromQL queries, you can modify the query.lookback-delta Prometheus option which is where this default 5m value is defined.

For example, specifying --query.lookback-delta=1d in your Prometheus launch options and restarting the service will cause the PromQL query my_metric to return the most recent value of my_metric looking back 24 hours.

Metrics outside this "look-back time window" are called stale.

pphysch
  • 169
  • 2
  • 4
0

Prometheus's instant query API can solve this.

Here is the example, you can see the timestamp in value is 1435781451.781.

$ curl 'http://localhost:9090/api/v1/query?query=up&time=2015-07-01T20:10:51.781Z'
{
   "status" : "success",
   "data" : {
      "resultType" : "vector",
      "result" : [
         {
            "metric" : {
               "__name__" : "up",
               "job" : "prometheus",
               "instance" : "localhost:9090"
            },
            "value": [ 1435781451.781, "1" ]
         },
         {
            "metric" : {
               "__name__" : "up",
               "job" : "node",
               "instance" : "localhost:9100"
            },
            "value" : [ 1435781451.781, "0" ]
         }
      ]
   }
}
54vault
  • 39
  • 4