kube-prometheus-stack / drop namespace

Question

i m trying to drop a namespace, stop prometheus to scrape anything from it. i have tried so many ways i got from the internet and the prometheus documentation. none seems to work.

i tried to set the 'prometheus.serviceMonitor.relabelings',

- source_labels: [__meta_kubernetes_pod_name, __meta_kubernetes_pod_container_name, __meta_kubernetes_namespace, namespace] separator: ; regex: (vicken|postgres)

and the generated config shows below, I still have data coming from that namespace. anyone knows how?

- job_name: serviceMonitor/ddhub-demo/my-kube-stack-kube-prometh-prometheus/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: kube-prometheus-stack-prometheus
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release]
    separator: ;
    regex: my-kube-stack
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_self_monitor]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: web
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: web
    action: replace
  - source_labels: [__meta_kubernetes_pod_name, __meta_kubernetes_pod_container_name,
      __meta_kubernetes_namespace, namespace]
    separator: ;
    regex: (vicken|postgres)
    replacement: $1
    action: drop
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    namespaces:
      names:
      - kube-demo`

score 0 · Answer 1 · answered Oct 04 '22 at 14:27

If you are using the Prometheus Operator, you cannot manually interfere with the prometheus config file. The Operator will overwrite your changes.

You will have success with using the serviceMonitorNamespaceSelector in your Prometheus definition. This will allow you to explicitly define the namespaces from which you want to pick up serviceMonitor objects. However, it is possible to have a cross reference, so if you additionally also explicitly specify targets appropriately in your serviceMonitor using the namespaceSelector you can make sure to only specify what you are interested in.

score 0 · Answer 2 · answered Apr 27 '23 at 15:00

For namespaces, what I've found if you're using the prometheus operator or the later kube-prometheus-stack chart is you need to relabel the namespace AND apply a metric relabel for the namespace you want to keep, e.g.:

kube-state-metrics:
  prometheus:
    monitor:
      metricRelabelings:
      - sourceLabels: [ namespace ]
        regex: my_namespace
        action: keep
      relabelings:
       - sourceLabels: [__meta_kubernetes_namespace]
         separator: ;
         regex: ^(.*)$
         targetLabel: namespace
         replacement: $1
         action: replace

Of course replace my_namespace with your target namespace. I've tried this with keep but I suspect it will work with drop as well.

kube-prometheus-stack / drop namespace

2 Answers2

Linked