How to add static labels to jobs in service monitors

Question

In prometheus we have the option to add a label to every metric of a job with something like this,

- job_name: 'your_job'                 
  honor_labels: true                         
  static_configs:                     
  - targets:                          
    - '127.0.0.1'          
    labels:                           
      cluster: 'stage'

I want to add labels to metrics but using servicemonitors. I am using blackbox prometheus operator to scan some websites. This is how my service monitor looks.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/instance: prometheus-blackbox-exporter
    app.kubernetes.io/name: prometheus-blackbox-exporter
    app.kubernetes.io/version: 0.20.0
    instance: primary
  name: prometheus-blackbox-exporter-example.com
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    metricRelabelings:
    - action: replace
      replacement: https://example.com
      sourceLabels:
      - instance
      targetLabel: instance
    - action: replace
      replacement: example.com
      sourceLabels:
      - target
      targetLabel: target
    params:
      module:
      - http_2xx
      target:
      - https://example.com
    path: /probe
    port: http
    scheme: http
    scrapeTimeout: 30s
  jobLabel: prometheus-blackbox-exporter
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      app.kubernetes.io/instance: prometheus-blackbox-exporter
      app.kubernetes.io/name: prometheus-blackbox-exporter

I want to add a label to the metrics coming from this job. The label is

project: monitoring

How can I do it using servicemonitors?

Answer 1

Add this under spec.endpoints[].metricRelabelings or spec.endpoints[].relabelings:

- targetLabel: project  # name
  replacement: monitoring  # value

e.g.:

spec:
  endpoints:
  - interval: 30s
    relabelings:
    - targetLabel: project
      replacement: monitoring

The option with spec.endpoints[].relabelings is better in your case because the label will appear not only on gathered metrics but on all automatic (such as up) as well.

Read more on relabeling in the docs: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config