0

While maintaining a simple set of a Grafana and a Prometheus pod, plus a few others, within a cluster on Azure Kubernetes Services (AKS)—I ran into an issue where a "instance" variable query set up in Grafana only retrieves any data from several, out of 52, VM's/instances in a Prometheus scrape job.

There are a couple other Grafana variables that can arbitrarily nest / be nested inside of the "instance" variable, but switching the order of them did nothing. the variables, including "instance" and "environment" which do not depend on each other and I tried switching

To elaborate, this is bizarre because I am getting perfect data for 11 out of the 52 VM's/instances while the others are not being populated in the "instance" Grafana variable as they should be. Maybe this is a backend issue, but have not found any oddities while probing around with kubectl.

Thank you!

Jackalakalaka
  • 91
  • 1
  • 7
  • can you confirm if all the 52 instances are sending the windows_iis_requests_total metric? – rohatgisanat May 31 '21 at 01:36
  • You nailed it on the head @rohatgisanat ! Even though all of the VMs/instances are up and running, each missing VM's Prometheus instance didn't contain the metric "windows_iis_requests_total" that was scanned for by the Grafana variable query and therefore was not included in the Grafana variable dropdown. Thanks a ton! – Jackalakalaka Jun 01 '21 at 11:03

1 Answers1

1

All the 52 instances were not sending the windows_iis_requests_total metric.Hence the label_values gave the incomplete response.

rohatgisanat
  • 708
  • 6
  • 15