How can I join two metrics in Prometheus where the label of a metric is part of the other metric label containing comma-separated values?

Question

After digging the doc and the web, maybe you'll be able to help me on this topic.

I am using Prometheus as a Data Source in Grafana.

We have a first metric, called module_status. I am filtering on it using a specific module name:

module_status{module_name="MODULE1"}

The raw instant result is:

module_status{dependencies="SERVICE_1,SERVICE_4,SERVICE_5", module_name="MODULE1"} 4

(4 is the actual value of the status = OK)

We have a second metric, called service_status. Raw instant results of the metric would be:

service_status{service_name="SERVICE_1"} 4
service_status{service_name="SERVICE_2"} 4
service_status{service_name="SERVICE_3"} 4
service_status{service_name="SERVICE_4"} 4
service_status{service_name="SERVICE_5"} 4

We need to display only the service_status where service_name is included in the label dependencies of module_status.

Many thanks for your help !

What I already tried: I created a variable module_name containing the module_name based on this query: label_values(module_status, module_name)

Then, I created another variable module_services containing the list of modules based on this query: label_values(module_status{module_name="$module_name"}, dependencies)

It works fine, since it returns me one line of CSV values, such as SERVICE_1,SERVICE_4,SERVICE_5

However, I was not able to create one line for each of these values in the variable...

Answer 1

There is a trivial way to do it. In PromQL, you can use a join with theon and group_left operators. In your case, the solution would be:

service_status * on(service_name) group_left(dependencies) module_status{module_name="MODULE1"}