Drop or rename scrape_config label

Question

I'd love to rename or drop a label from a /metrics endpoint within my metric. The metric itself is from the kube-state-metrics application, so nothing extraordinary. The metric looks like this:

kube_pod_container_resource_requests{container="alertmanager", instance="10.10.10.10:8080", funday_monday="blubb", job="some-kube-state-metrics", name="kube-state-metrics", namespace="monitoring", node="some-host-in-azure-1234", pod="alertmanager-main-1", resource="memory", uid="12345678-1234-1234-1234-123456789012", unit="byte"} 209715200

The label I'd love to replace is instance because it refers to the host which runs the kube-state-metrics application and I don't care about that. I want to have the value of node in instance and I've been trying so for hours now and can't find a way - I wonder if it's not possible at all!?

The way I'm grabbing the /metrics endpoint is through the means of a scrape-config which looks as follows:

- job_name: some-kube-state-metrics
      scrape_interval: 30s
      scrape_timeout: 10s
      metrics_path: /metrics
      kubernetes_sd_configs:
      - api_server: null
        role: pod
      scheme: http
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_labelpresent_kubeStateMetrics]
        regex: true
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_name]
        regex: (.*)
        replacement: $1
        target_label: name
        action: replace
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        separator: ;
        regex: http
        replacement: $1
        action: keep
      - source_labels: [node]
        regex: (.*)
        replacement: blubb
        target_label: funday_monday
        action: replace
      - action: labeldrop
        regex: "unit=(.*)"
      - source_labels: [ __name__ ]
        regex: 'kube\_pod\_container\_resource\_requests'
        action: drop

As you can tell, I've been trying to drop labels as well, namely the unit-label (just for testing purposes) and I also tried to drop the metrics all together. The funday_monday is an example that changed because I wanted to know if static relabels are possible (it works!) - before it looked like this:

- source_labels: [node]
        regex: (.*)
        replacement: $1
        target_label: funday_monday
        action: replace

Help is appreciated.

anemyte · Accepted Answer · 2022-02-18T16:15:04.563

The problem is that you are doing those operations at the wrong time. relabel_configs happens before metrics are actually gathered, so, at this time, you can only manipulate the labels that you got from service discovery.

That node label comes from the exporter. Therefore, you need to do this relabeling action under metric_relabel_configs:

metric_relabel_configs:
- source_labels: [node]
  target_label: instance

Same goes for dropping metrics. If you wish a bit more info, I answered a similar question here: prometheus relabel_config drop action not working

Drop or rename scrape_config label

1 Answers1