0

What can be a reliable way of clearing stale data from PushGateway for an app that has retired (e.g. instance is shut down on unusual circumstances) w/o using manual involvement?

Michael Doubez
  • 5,937
  • 25
  • 39
Divs
  • 1,578
  • 2
  • 24
  • 51

2 Answers2

1

you can clear / remove app's metric data/ job by using below url

job name: sample_job

curl --location --request DELETE 'http://localhost:9091/metrics/job/sample_job'

success code - 202

if you are running multiple instances of same app, you can delete individual instance's metric also,

job name: sample_job

and

InstnaceId = 20148

curl --location --request DELETE 'http://localhost:9091/metrics/job/sample_job/instance/20148'

success code - 202

but the only problem could be, If the app delete its own metric from the pushGateway upon its exit, prometheus may lose it. Yes, if the app exits between the prometheus scraping the pushgateway.

Shankar S
  • 387
  • 5
  • 13
-1

The Pushgateway is for service-level batch jobs, which are independent are individual serving instances that can go down. What you probably want to do is scrape this instance directly using Prometheus, rather than trying to convert Prometheus to push.

I'd recommend reading https://prometheus.io/docs/practices/pushing/

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
  • Thanks! Problem is, this is a work-around as the Prometheus sits outside the allowable network and can't access my app directly. So pushing the metrics to PushGateway by the apps and Prometheus scrapping the PushGateway seems to be the only way around for now. – Divs Jan 11 '18 at 13:14
  • It's recommended to run Prometheus directly beside what it is scraping, which avoids that issue. – brian-brazil Jan 11 '18 at 15:43