Grafana - Mean calculation from dynamic panels inside a dashboard

Question

We have a Grafana dashboard with Prometheus as data source. Inside the dashboard, we have a row that will repeat based on the service(multi select variable) selected. The repeated rows will have panels for SLI mean.

My requirement is: I have to find the Mean for all the services(selected) in a single panel.

Here test-service, test-service-1 are the values selected from the service variable(multi select). One of the panel inside each row is 5min SLI mean. Below attached image will show the 5min SLI panel

Inside General Row I need a similar panel, But here SLI calculation should be the mean value of all the other 5min SLI panel.

Say, if 2 services are selected, we will have 2 rows as shown in the attached image. The 5m SLI(inside General row) should be the mean of those two services selected. If 2 more services selected(total 4 service), the 5m SLI panel inside General row should change by calculating mean from 4 other 5min SLI panels.

Note: I am new to Grafana, Sharing some steps/ideas will be appreciated.

Answer 1

Prometheus' query language, PromQL, has an avg function which will likely do what you want here. Check out https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators for extended documentation on similar functions.

Let's assume your metric is called sli. When you include a Prometheus query sli in your Grafana graph, Prometheus picks every instance of that metric and Grafana then displays them.

You can (and I assume you have?) filter this by having a variable in Grafana, and doing a PromQL label selector on that variable, like such: - sli{service=~"${service}"}.

What this actually is getting you, though, is a set of time series from Prometheus, which Grafana is handily converting into a chart for you. But you can ask Prometheus to average it for you before Grafana gets its hands on the result:

avg(sli{service=~"${service}"})

This will return one single time series (instead of many of them) representing the average of the time series selected by sli{service=~"${service}"} at every point in time.

I've had to make some assumptions here since you didn't show your query in your initial question; feel free to update your question with more details and drop a comment here if you want me to update the answer.