Prometheus gives empty query result although individual component metrics are available

Question

In a Grafana dashboard a custom metric is shown. This metric is received via Prometheus. The metric is built via Spring Boot Actuator / Micrometer.

In about half of the cases the Prometheus query of the metric gives an empty result. In the other cases the value is valid. So, it may have to do with a threshold of scraping?

Analysing the query Prometheus uses, an empty result is presented:

http://valid.url/prometheus/api/v1/query?query=last_seconds_since_startime_seconds&time=1645521095.864

http://valid.url/prometheus/api/v1/query?query=last_seconds_since_startime_seconds

The result is:

{"status":"success","data":{"resultType":"vector","result":[]}}

Using this query a complete result is shown (after about 17 seconds) with the actual value:

http://valid.url/stackname/componentname/actuator/prometheus

In the text is:

last_seconds_since_startime_seconds 63.0

The environment is Spring Boot, Micrometer, Actuator, Prometheus and Grafana. The actuator query is built with Micrometer:

Gauge.builder( LAST_SECONDS_SINCE_STARTIME, this,
            PrometheusStatistics::secondsSinceStart)
            .description("Seconds since last fetch")
            .baseUnit("seconds")
            .strongReference(true)
            .register(meterRegistry);

tm1701 · Accepted Answer · 2022-02-24T12:03:23.440

The solution was to update the timeout for the scraping in the prometheus.yml file for specific components in the Docker Swarm.

The global setting is left to the default of 10s.

For the specific component:

- job_name: 'my-component'
  metrics_path: /actuator/prometheus
  scrape_interval: 30s
  scrape_timeout: 25s    <== this one
  dns_sd_configs:
    - names:
        - 'tasks.mycomponent'
      type: 'A'
      port: 8080

Prometheus gives empty query result although individual component metrics are available

1 Answers1