0

I have a Openshift cluster and want to enable annotation-based scrape for Prometheus for openshift-user-workload-monitoring. I.e. I want Prometheus to scrape any pod annotated as follows:

    prometheus.io/port: "9100"
    prometheus.io/scrape: "true"

I tried configuring Prometheus as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheusOperator:
      logLevel: debug
    prometheusK8s:
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: (.+):(?:\d+);(\d+)
          replacement: ${1}:${2}
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name

but didn't work so far.

One of the solution is using ServiceMonitor or PodMonitor, but I would really like to get to understand what is failing here.

AntMor
  • 417
  • 6
  • 18
  • 1
    I suggest checking logs and `Status->Targets` (in Prometheus web UI) for any errors. If the latter does not have any items under your `'kubernetes-pods'` job, then it's probably because Prometheus can't find any pods with `prometheus.io/scrape: true` annotation. Try changing `regex: true` to `regex: "true"` because `true` without quotes is a boolean in YAML. – anemyte Jul 05 '23 at 18:23

0 Answers0