3

Context

We have a Spring Boot application, deployed into K8s cluster (with 2 instances) configured with Micrometer exporter for Prometheus and visualization in Grafana.

My custom metrics

I've implemented couple of additional Micrometer metrics, that report some information regarding business data in the database (PostgreSQL) and I could see those metrics in Grafana, however separately for each pod.

Problem:

For our 2 pods in Grafana - I can see separate set of same metrics and the most recent value can be found by choosing (by label) one of the pods.

However there is no way to tell which pod reported the most recent values.

Is there a way to somehow always show the metrics values from the pod that was scraped last (ie it will contain the most fresh metric data)?

Right now in order to see the most fresh metric data - I have to switch pods and guess which one has the latest values.

(The metrics in question relate to database, therefore yielding the same values no matter the pod from which they are requested.)

antoni.rasul
  • 648
  • 6
  • 12

1 Answers1

4

In Prometheus, you can obtain the labels of the latest scrape using topk() and timestamp() function:

topk(1,timestamp(up{job="micrometer"}))

This can then be used in Grafana to populate a (hidden) variable containing the instance name:

Name: instance
Type: Query
Query: topk(1,timestamp(up{job="micrometer"}))
Regex: /.*instance="([^"]*)".*/

I advise to active the refresh on time range change to get the last scrape in your time range.

Then you can use the variable in all your dashboard's queries:

micrometer_metric{instance="${instance}"}

EDIT: requester wants to update it on each data refresh

If you want to update it on each data refresh, it needs to be used in every query of your dashboard using AND logical operator:

micrometer_other_metric AND ON(instance) topk(1,timestamp(up{job="micrometer"}))

vector1 AND vector2 results in a vector consisting of the elements of vector1 for which there are elements in vector2 with exactly matching label sets. Other elements are dropped.

Michael Doubez
  • 5,937
  • 25
  • 39
  • Thanks! Marked as accepted answer, since indeed this query should determine the instance with latest timestamp. However, I ended up by taking max(..) from my metrics, which is not ideal, but gets the job done (since I am more concerned of seeing when value goes up than when it falls). The reasons I went with this "simplified" approach are: – antoni.rasul Dec 30 '21 at 20:29
  • - Receiving when trying to add variable: "Error updating options: cannot parse "topk(1,timestamp(up{job=\"micrometer\"}))": expecting metricSelector; got "topk(1, timestamp(up{job=\"micrometer\"}))" - Even if it allowed me to use operators - the variable would be updated at most on timeframe change, and we need it with each data refresh. – antoni.rasul Dec 30 '21 at 20:35
  • I tried also to pack this query directly into Panel query, but no luck - since I couldn't find a way to create a query that will use dynamically calculated label value. In general - it looks like Prometheus lacks some function, like 'last_scraped(..)' that would aggregate arguments by taking the one with latest timestamp() of scraping. – antoni.rasul Dec 30 '21 at 20:35
  • Using `max()` won't work as you will loose the label of the instance that was last scaped. Also note that metrics going up and down won't affect that: topk is applied on the timestamp of the metric, not the value (and timestamp should be always increasing). – Michael Doubez Dec 31 '21 at 08:14
  • I completed the answer with a way to apply that on each refresh. – Michael Doubez Dec 31 '21 at 08:22
  • For me when creating this variable I also had to wrap it in `query_result(topk(1,somethingHere))` – waeewoae Feb 16 '22 at 07:12