Prometheus Operator not scraping colocated etcd metrics

Question

I have a K8s cluster with colocated etcd deployed on-prem servers, using Kubespray. I don't see the etcd metrics getting scraped by Prometheus operator. Prometheus operator deployed using helm v3.5.4. K8s version 1.22 , Helm chart prometheus-community/kube-prometheus-stack version 25.0.0 , 3 node control plane on CentOS 7.

Prometheus config shows a job for etcd - job_name: serviceMonitor/monitoring/kube-prometheus-kube-prome-kube-etcd/0 . But there is no service for etcd in the list of Services for Prometheus.

There are no endpoints defined for etcd

Values.yml (updated with volumes ) for helm deployment

prometheus:
    service: 
        type: NodePort
        externalTrafficPolicy: Local
    ingress:
        enabled: true
        annotations:
            kubernetes.io/ingress.class: "custom"
        hosts:
            - prometheus.{{ cluster_domain }}.mydomain.com
        paths: 
            - /
        pathType: Prefix
        tls:
            - secretName: 
    prometheusSpec:
        storageSpec:
            volumeClaimTemplate:
                spec:
                    accessModes: ["ReadWriteOnce"]
                    storageClassName: rook-ceph-block
                    resources:
                        requests:
                            storage: {{ monitoring.storage_size }}
        volumeMounts:
        - name: cert-vol
          mountPath: "/etc/prometheus/secrets/etcd-certs"
          readOnly: true

        volumes:
        - name: cert-vol
          secret: 
            secretName: etcd-certs

kubeEtcd:
    enabled: true
    endpoints:
      - 172.1.1.1
      - 172.1.1.2
      - 172.1.1.3
  
    service:
        port: 2379
        targetPort: 2379
    serviceMonitor:
        scheme: https
        insecureSkipVerify: true
        caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
        certFile: /etc/prometheus/secrets/etcd-certs/client.crt
        keyFile: /etc/prometheus/secrets/etcd-certs/client.key

etcd is fine, I can get the metrics from etcd if I curl to it. Prometheus is not even scraping the metrics. do I need to look for something specific in etcd logs ? — RDs, Jan 24 '22 at 15:22
I noticed a problem , in the prometheus pod logs, prometheus cannot find the path to the certs. — RDs, Jan 24 '22 at 15:28
I fixed the missing path issue to the certs and updated the question with the info. but still missing etcd data. — RDs, Jan 24 '22 at 23:21
Nothing indicating in Prometheus logs, that it failed to collect etcd metrics. — RDs, Jan 26 '22 at 21:50

score 1 · Answer 1 · edited Jan 31 '22 at 08:00

I added the endpoints to kubeEtcd section to get it to work. The updated values.yaml is like below (changed IP adresses):

prometheus:
    service: 
        type: NodePort
        externalTrafficPolicy: Local
    ingress:
        enabled: true
        annotations:
            kubernetes.io/ingress.class: "custom"
        hosts:
            - prometheus.{{ cluster_domain }}.mydomain.com
        paths: 
            - /
        pathType: Prefix
        tls:
            - secretName: 
    prometheusSpec:
        storageSpec:
            volumeClaimTemplate:
                spec:
                    accessModes: ["ReadWriteOnce"]
                    storageClassName: rook-ceph-block
                    resources:
                        requests:
                            storage: {{ monitoring.storage_size }}
        volumeMounts:
        - name: cert-vol
          mountPath: "/etc/prometheus/secrets/etcd-certs"
          readOnly: true

        volumes:
        - name: cert-vol
          secret: 
            secretName: etcd-certs

kubeEtcd:
    enabled: true
    endpoints:
      - 172.1.1.1
      - 172.1.1.2
      - 172.1.1.3
  
    service:
        port: 2379
        targetPort: 2379
    serviceMonitor:
        scheme: https
        insecureSkipVerify: true
        caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
        certFile: /etc/prometheus/secrets/etcd-certs/client.crt
        keyFile: /etc/prometheus/secrets/etcd-certs/client.key

Prometheus Operator not scraping colocated etcd metrics

1 Answers1