1

I was wondering is there any way to filter by Thanos store in Thanos+prometheus+Grafana cluster. I am trying to query from Grafana dashboard to get results from exactly one store (we have 3). Any help is appreciated, Thank you in advance

Behzod Olloqulov
  • 11
  • 4

2 Answers2

1

I was able to do it by using following query: prometheus_engine_queries with following filter:

/.*env=\"(.*)\",instance/

env - this is from -set storeGateway.additionalLabels.env and it is added to every metrics

Rolandas
  • 11
  • 1
  • Thank you for your reply. As I am new in PromQL and Grafana could you please explain it in more detail? Thank you in advance – Behzod Olloqulov Apr 01 '21 at 12:26
  • You should modify configs so thanos sidecars would have Announced LabelSets based on source. This should add those labels to every metric you store in thanos. Later just modify dashboards to use those labels as variables for selection and in expressions – Rolandas Apr 01 '21 at 20:36
0

I experienced a similar issue. I wasn't able to filter on any of my externalLabels in Grafana PromQL queries :

prometheus:
  externalLabels:
    cluster: my_cluster

My misstake was that I configured the data source wrong in Grafana. I used prometheus-operated instead of the actual query service :

http://thanos-query:9090
Ghis
  • 845
  • 10
  • 16