How to configure a Grafana dashboard for multiple Prometheus datasources?

Question

I'm trying to configure a Grafana dashboard for multiple Prometheus datasources. These two datasources are present in two different servers.

Prometheus scrape the metrics from three jobs, prometheus itself, node-exporter for host metrics, advisor for docker containers metrics.

This is what prometheus.yaml looks like:

global:
 scrape_interval: 5s
scrape_configs:
 - job_name: 'prometheus'
   static_configs:
       - targets: ['localhost:9090']
 - job_name: 'node-exporter'
   static_configs:
     - targets: ['node-exporter:9100']
 - job_name: 'cadvisor'
   static_configs:
     - targets: ['cadvisor:8080']

The same yaml configuration is being used in both the servers.

I created two datasources in Grafana UI and their respective names are Prometheus, and Prometheus-Certify. The first datasource is the default one. I'm using Grafana v7.1.0.

This is what my dashboard looks like for the datasource Prometheus:

There are other panels as well, but showing these panels would be enough to explain the issue.

If I click on the Host dropdown option, then I can see the hosts cadvisor:8080, localhost:9090, and node-exporter:9100.

Requirement:

What I'm trying to achieve is to configure this dashboard for both the data sources. I can see all the options in the datasource dropdown menu default, Prometheus, Prometheus-Certify. I want the dashboard to show me the data from a particular source when I select that particular datasource. And this is where I'm getting stuck.

I did my research on this and found that we can show data from two different sources at the same time in a panel. I saw this option in the Grafana documentation and also this thread which actually gets me closer to what I'm trying to achieve.

So I followed this approach and configured --Mixed-- datasources for a panel.

It worked flawlessly and performing the same operation on different panels resulted in this:

Though it worked but, this is not what I was trying to do. The problems here are:

1. I'll have to do this for every panel in order to see data from both the sources.
2. It shows data from both sources at the same time. I want to see data from the selected datasource only, not both.
3. If I add another datasource in the future, I'll have to perform all the operations all over again.

I saw [templates and variables][6] being suggested in many places. I gave it a shot but it didn't do anything.

If changing variables is the way to go then please provide suggestions specific to this issue only as I've already gone through that section and couldn't find anything specific to this.

I hope I was able to explain my problem to you people. I just want to see the data from the selected datasource without creating a new dashboard. Your help would be very much appreciated. Thank you.

Answer 1

For me disabling a default Prometheus datasource did the trick. Without a default data source you can select the Prometheus datasource in the query.

Answer 2

I realized that the dashboard I imported creates a variable DS_PROMETHEUS, and its type is Datasource.

This variable holds the value of the selected datasource. I was unable to see the data from the selected datasource because the datasource was kinda hardcoded in the panels' queries.

As you can see in the image above the preselected datasource is Prometheus. Changing this value to $DS_PROMETHEUS did the job.

So it all boiled down to how do I make this variable as the datasource for every query in every panel? The question now was how to run a query/command to modify settings of multiple panels in one go.

After doing some research I found this thread. It has been categorized as a feature request. Person/People have already developed a tool to handle these kinds of scenarios. As of now, I haven't used this tool but I'll update my answer once I use it.

The simple and the old-school way was to replace prometheus with $DS_PROMETHEUS everywhere in the dashboard's JSON. So I did that and Voila! it worked.