1

I have two "slave" prometheus severs, one in each of my kubernetes clusters. I have one centralised prometheus for federation and alerting.

Sometimes, it happens that a "slave" stops delivering metrics. How to detect it? How to create an alert that catches such a situation.

Unfortunately, prometheus always sees its federated peers as UP. No matter what.

Skarab
  • 6,981
  • 13
  • 48
  • 86

1 Answers1

-1

We need a bit more information here. If up is 1 then everything is okay. The real question is why you're getting a successful scrape with no data. Have you tried debugging that?

I'd also suggest alerting as deep in the stack as you can, see https://www.robustperception.io/federation-what-is-it-good-for/

brian-brazil
  • 31,678
  • 6
  • 93
  • 86