13

Suppose we are collecting the same metrics for one month and now we want to modify the metrics to have extra label (in the old data as well), how can we do that. Existing metric: 

mongodb_exporter_last_scrape_duration_seconds{instance="127.0.0.1:9216",job="mongo"}

Want to change that to:

mongodb_exporter_last_scrape_duration_seconds{cluster="stage", instance="127.0.0.1:9216",job="mongo"}
Nipun Talukdar
  • 4,975
  • 6
  • 30
  • 42

5 Answers5

14
- job_name: 'your_job'                 
  honor_labels: true                         
  static_configs:                     
  - targets:                          
    - '127.0.0.1'          
    labels:                           
      cluster: 'stage'
Tonny Madsen
  • 12,628
  • 4
  • 31
  • 70
k0ste
  • 157
  • 2
  • 15
    While this code snippet may be the solution, including an explanation really helps to improve the quality of your post. Remember that you are answering the question for readers in the future, and those people might not know the reasons for your code suggestion – Rahul Gupta Dec 29 '17 at 11:30
  • 1
    That will add the label "stage" to the newly scraped data. But I want to add that label to previously collected data points as well. – Nipun Talukdar Dec 29 '17 at 19:57
  • This is for static configurations, if i have to add a label to my kubernetes_sd_configs , where should i add this additional label? – Sathish Kumar Mar 30 '20 at 04:39
  • Please give explanation of your snippet! – Vaiaro May 17 '22 at 13:59
9

Unfortunately, it is not possible to change the labels on old metrics in Prometheus.

The storage is only updated by new scrapes and then it becomes immutable.

SuperQ
  • 116
  • 1
  • 1
5

if you have a service discovery config, you can also add labels in the relabels_config

relabel_configs:
  - target_label: cluster
    replacement: stage
mkl
  • 51
  • 1
  • 1
4

Prometheus doesn't provide the ability to add new labels to historical time series. But it is possible to add missing labels during queries with label_replace() function:

label_replace(
  mongodb_exporter_last_scrape_duration_seconds,
  "cluster",
  "staging",
  "cluster",
  ""
)

This label_replace() call adds cluster="staging" label only if the cluster label is missing in the original time series selected with mongodb_exporter_last_scrape_duration_seconds series selector.

valyala
  • 11,669
  • 1
  • 59
  • 62
0

https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace https://medium.com/@texasdave2/replace-and-remove-a-label-in-a-prometheus-query-9500faa302f0

In fact, the label_replace method only add a new label to metrics but not replace. So I can use this method to add a new label if you do not want to change prometheus.yml config.

For your case:

mongodb_exporter_last_scrape_duration_seconds{instance="127.0.0.1:9216",job="mongo"}

label_replace(mongodb_exporter_last_scrape_duration_seconds{instance="127.0.0.1:9216",job="mongo"}, "cluster", "stage", "(.*)")

lcgogo
  • 95
  • 1
  • 5