How to scrape all metrics from a federate endpoint?

Question

We have a hierachical prometheus setup with some server scraping others. We'd like to have some servers scrape all metrics from others.

Currently we try to use match[]="{__name__=~".*"}" as a metric selector, but this gives the error parse error at char 16: vector selector must contain at least one non-empty matcher.

Is there a way to scrape all metrics from a remote prometheus without listing each (prefix) as a match selector?

score 24 · Accepted Answer · answered Aug 31 '16 at 15:42

24

Yes, you can do: match[]="{__name__=~".+"}" (note the + instead of * to not match the empty string).

Prometheus requires at least one matcher in a label matcher set that doesn't match everything.

answered Aug 31 '16 at 15:42

Julius Volz

688
4
6

3

I found __name__=~"..*" worked okay on prom 2.0.0. Careful with the unnecessary " outside the {}. – fche Mar 07 '18 at 23:26

score 15 · Answer 2 · answered Sep 07 '18 at 11:00

15

I have tried multiple example, even the ones from prometheus docs but nothing worked.

Instead this works for me

http://prometheus-ip:9090/federate?match[]={job!=""}

answered Sep 07 '18 at 11:00

Radu Toader

1,521
16
23

2

some recording rule metric not has label job – wow qing Nov 09 '19 at 02:58

score 6 · Answer 3 · answered Aug 31 '16 at 17:52

6

Federation is not intended to transfer all metrics, and if you try to do so you'll eventually run into problems.

Instead, aggregate up the metrics you want and then federate only those.

answered Aug 31 '16 at 17:52

brian-brazil

31,678
6
93
86

5

Thanks for the warning. I'll keep that in mind when we run into trouble, but for now I'm sure scraping all metrics is going to be fine. We'll come back to pre-aggreation when things settle and we have a better understanding of what we need to look up for and what not. – tex Sep 01 '16 at 07:49

score 2 · Answer 4 · answered Nov 21 '18 at 19:09

2

Here is mine config file.

With this limits you can filter from which exporters (jobs) to scrape:

  params:
    'match[]':
    - '{job=~"node-exporter|kube-state|fluentbit"}'
    - '{__name__=~"job:.*"}'

answered Nov 21 '18 at 19:09

SQERISON

21
1

score 1 · Answer 5 · answered Jun 19 '19 at 07:58

1

Add these in your federation job

    params:
      match[]:
        - '{__name__=~".+"}'
        - '{__name__=~"^job:.*"}'
        - '{job="prometheus"}'
        - '{job="node"}'
        - '{__name__="server_labels"}'

answered Jun 19 '19 at 07:58

Sachin Arote

907
13
22

score 1 · Answer 6 · answered Jul 19 '20 at 15:25

I was trying to reach the same config, and after trying multiple solutions, the one that actually worked for me was:

params:
  match[]:
    - '{job=~".+"}'

This simple settings collected all the metrics which had job. The same didn't work with {__name__=~"job:.*"}.

How to scrape all metrics from a federate endpoint?

6 Answers6