How to rename label within a metric in Prometheus

Question

I have a query:

node_systemd_unit_state{instance="server-01",job="node-exporters",name="kubelet.service",state="active"} 1

I want the label name being renamed (or replaced) to unit_name ONLY within the node_systemd_unit_state metric. So, desired result is:

node_systemd_unit_state{instance="server-01",job="node-exporters",unit_name="kubelet.service",state="active"} 1

There are many other metrics with a label name name in the node-exporters job. That's why I can't use relabel config across the job.

score 33 · Answer 1 · edited Feb 22 '19 at 22:59

33

you can use the label_replace function in promQL, but it also add the label, don't replace it

label_replace(
  <vector_expr>, "<desired_label>", "$1", "<existing_label>", "(.+)"
)

label_replace(
node_systemd_unit_state{instance="server-01",job="node-exporters",name="kubelet.service",state="active"},
"unit_name","$1","name", "(.+)"
)

So, to avoid the repetition you can add:

sum(label_replace(
    node_systemd_unit_state{instance="server-01",job="node-exporters",name="kubelet.service",state="active"},
    "unit_name","$1","name", "(.+)"
    )
)by(unit_name)

edited Feb 22 '19 at 22:59

brooksrelyt

3,925
5
31
54

answered Feb 22 '19 at 18:35

Chus

926
8
9

3

this is the method without doing any configuration change! – Deepak Deore Jul 01 '19 at 11:51
What if I want to change the actual name of the metric (e.g. __name__) rather than a regular label. – jayarjo Jul 27 '22 at 19:14

score 16 · Answer 2 · answered Mar 27 '20 at 21:42

I got tired of all the fragmented documentation and I feel I provided a better answer in this post here: https://medium.com/@texasdave2/replace-and-remove-a-label-in-a-prometheus-query-9500faa302f0

Replace is not a true REPLACE

Your goal is to simply replace the old label name “old_job_id” with a new label name “new_task_id”. Prometheus label_replace will really “add” the new label name. It will preserve the old label name as well… So, that could be a problem, it’s not a true “replace in place”.

So if you want to “add” your new label name and “remove” the old label name, you need to do this:

sum without (old_job_id) (label_replace(metric, "new_task_id", "$1", "old_job_id", "(.*)"))

Here’s how this reads:

sum without (old_job_id) will remove the old label name from the query output
metric is your metric, like “node_filesystem_avail_bytes”
“new_task_id” is where you would put your new label name
“$1” is regex for using the string in new label name, don’t change this
“old_job_id” is where you’ll put your old label, the one you want to get rid of (.*……. that mess is regex that will replace the whole label name

score 7 · Accepted Answer · answered Jan 17 '19 at 13:09

7

You can have more than one source label, so:

- source_labels: [__name__, name]
  regex: "node_systemd_unit_state;(.+)"
  target_label: unit_name

If the label name isn't matching up with other metrics/exporters, you should file a bug with them. Relabelling like this should only be a temporary solution while a proper fix is being pursued.

answered Jan 17 '19 at 13:09

brian-brazil

31,678
6
93
86

1

great! But in this case I'll always have two labels: `name` and `unit_name` which values is the same every time. I'd like to avoid this kind of duplication. – Konstantin Vustin Jan 17 '19 at 14:00
There doesn't appear to be a way to do this, apart from temporarily storing the value of `name` in a `__tmp_name` label, dropping it from all metrics and then adding it back to all metrics except `node_systemd_unit_state`. But writing the regex to match everything but `node_systemd_unit_state` will be a PITA and the whole process will probably not be all that efficient. – Alin Sînpălean Jan 18 '19 at 14:25

valyala · Answer 4 · 2022-07-11T12:53:42.360

Prometheus allows renaming labels in the following places:

During metric scrape. In this case you need to add the following relabeling rules into metric_relabel_configs section at the corresponding scrape_config:

metric_relabel_configs:
- source_labels: [__name__, name]
  regex: "node_systemd_unit_state;(.+)"
  target_label: unit_name
- source_labels: [__name__, name]
  regex: "node_systemd_unit_state;(.+)"
  target_label: name
  replacement: ""

The first rule copies name label value into unit_name label for metrics with node_systemd_unit_state name. The second rule sets name label value to an empty string (e.g. deletes the name label) for metrics with node_systemd_unit_state name.

During querying via label_join() function:

label_join(
  label_join(node_systemd_unit_state, "unit_name", "", "name"),
  "name", "", "non_existing_label"
)

The inner label_join() copies name label into unit_name label. The outer label_join() substitutes the original name label with an empty string (e.g. remove the name label).

As you can see, label_join() isn't the best function for label renaming. The label_replace() isn't the best function for label renaming too. While Prometheus doesn't provide better solutions for label renaming, such solution exist in Prometheus-like systems such as VictoriaMetrics (I'm the author of this system). It provides label_move() function:

label_move(node_systemd_unit_state, "name", "unit_name")

Additionally, VictoriaMetrics provides if option for conditional relabeling rules. For example, the following relabeling rules are equivalent to the rules above, but are easier to understand and maintain:

metric_relabel_configs:
- if: 'node_systemd_unit_state{name!=""}'
  source_labels: [name]
  target_label: unit_name
- if: 'node_systemd_unit_state{name!=""}'
  target_label: name
  replacement: ""

You should have disclaimed that you are the CTO of VictoriaMetrics. — nouney, Jul 11 '22 at 08:51

score 0 · Answer 5 · answered Nov 11 '22 at 05:34

https://demo.promlens.com/?q=label_replace(up%2C%20%22hostname%22%2C%20%22%241%22%2C%20%22instance%22%2C%20%22(.%2B)%3A(%5C%5Cd%2B)%22)

Assume you have metric namely up

up{instance="cadvisor:8080", job="cadvisor"}    
up{instance="demo-service-0:10000", job="demo"} 
up{instance="demo-service-1:10001", job="demo"} 
up{instance="demo-service-2:10002", job="demo"} 
up{instance="docker-hub-exporter:9170", job="docker-hub-exporter"}  
up{instance="node-exporter:9100", job="node"}   
up{instance="prometheus:9090", job="prometheus"}

Now query
label_replace(up, "hostname", "$1", "instance", "(.+)-(.+)-(\\d+):(\\d+)")
will return with new label hostname with first match group ("$1").

up{instance="cadvisor:8080", job="cadvisor"}    
up{hostname="demo", instance="demo-service-0:10000", job="demo"}
up{hostname="demo", instance="demo-service-1:10001", job="demo"}
up{hostname="demo", instance="demo-service-2:10002", job="demo"}
up{instance="docker-hub-exporter:9170", job="docker-hub-exporter"}
up{instance="node-exporter:9100", job="node"}
up{instance="prometheus:9090", job="prometheus"}

Query
label_replace(up, "hostname", "$2", "instance", "(.+)-(.+)-(\\d+):(\\d+)")
will return with new label hostname with second match group ("$2") of the regEx.

up{instance="cadvisor:8080", job="cadvisor"}    
up{hostname="service", instance="demo-service-0:10000", job="demo"} 
up{hostname="service", instance="demo-service-1:10001", job="demo"} 
up{hostname="service", instance="demo-service-2:10002", job="demo"} 
up{instance="docker-hub-exporter:9170", job="docker-hub-exporter"}  
up{instance="node-exporter:9100", job="node"}   
up{instance="prometheus:9090", job="prometheus"}

How to rename label within a metric in Prometheus

5 Answers5

Linked