k8s dashboard metric does not work (Metric client health check failed)

Question

In my case, dashboard-metrics-scraper does not work. Is there any suggestion?

My test environment is kubernetes 1.20.4.
kubectl top nodes/pods are works fine

Following commands do NOT work

on kubernetes-dashboard log

2021/03/22 06:58:35 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2021/03/22 06:59:05 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.

on console

with adding /proxy/healthz goes bad request. If it is removing this (/proxy/healthz), it works fine.

$ kubectl get --raw "/api/v1/namespaces/kubernetes-dashboard/services/dashboard-metrics-scraper/proxy/healthz"
Error from server (BadRequest): the server rejected our request for an unknown reason

Following commands WORK FINE

wget from container

kubectl exec -it busybox -- wget --spider http://dashboard-metrics-scraper.kubernetes-dashboard.svc.cluster.local:8000
Connecting to dashboard-metrics-scraper.kubernetes-dashboard.svc.cluster.local:8000 (10.110.73.42:8000)

kubectl proxy seems work, but it is redirected.

$ curl http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/http:dashboard-metrics-scraper:/proxy/healthz
<HTML><HEAD>
<TITLE>Redirect</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Redirect (authentication_redirect_to_virtual_host)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
You are being redirected to the authentication virtual host.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">

</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>

</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>

kubectl get --raw to dashboard-metrics-scraper works fine (but if it is adding /proxy/healthz, it does not work.

$ kubectl get --raw "/api/v1/namespaces/kubernetes-dashboard/services/dashboard-metrics-scraper"
{"kind":"Service","apiVersion":"v1","metadata":{"name":"dashboard-metrics-scraper","namespace":"kubernetes-dashboard","uid":"43f83d61-81b2-4d57-8321-516158444cd1","resourceVersion":"6033988","creationTimestamp":"2021-03-22T02:53:15Z","labels":{"k8s-app":"dashboard-metrics-scraper"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"dashboard-metrics-scraper\"},\"name\":\"dashboard-metrics-scraper\",\"namespace\":\"kubernetes-dashboard\"},\"spec\":{\"ports\":[{\"port\":8000,\"targetPort\":8000}],\"selector\":{\"k8s-app\":\"dashboard-metrics-scraper\"}}}\n"},"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"v1","time":"2021-03-22T02:53:15Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:k8s-app":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":8000,\"protocol\":\"TCP\"}":{".":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{".":{},"f:k8s-app":{}},"f:sessionAffinity":{},"f:type":{}}}}]},"spec":{"ports":[{"protocol":"TCP","port":8000,"targetPort":8000}],"selector":{"k8s-app":"dashboard-metrics-scraper"},"clusterIP":"10.110.73.42","clusterIPs":["10.110.73.42"],"type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}}

dashboard-metrics-scraper service seems work

$ kubectl describe svc -n kubernete
s-dashboard dashboard-metrics-scraper
Name:              dashboard-metrics-scraper
Namespace:         kubernetes-dashboard
Labels:            k8s-app=dashboard-metrics-scraper
Annotations:       <none>
Selector:          k8s-app=dashboard-metrics-scraper
Type:              ClusterIP
IP Families:       <none>
IP:                10.110.73.42
IPs:               10.110.73.42
Port:              <unset>  8000/TCP
TargetPort:        8000/TCP
Endpoints:         172.16.8.142:8000
Session Affinity:  None
Events:            <none>

score 1 · Answer 1 · edited Nov 18 '22 at 12:42

I solved it: screen Add hostNetwork: true in deploy dashboard-metrics-scraper k8s: v1.25.4

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: dashboard-metrics-scraper
  template:
    metadata:
      labels:
        k8s-app: dashboard-metrics-scraper
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: dashboard-metrics-scraper
          image: kubernetesui/metrics-scraper:v1.0.8
          ports:
            - containerPort: 8000
              protocol: TCP
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /
              port: 8000
            initialDelaySeconds: 30
            timeoutSeconds: 30
          volumeMounts:
          - mountPath: /tmp
            name: tmp-volume
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      serviceAccountName: kubernetes-dashboard
      hostNetwork: true
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      volumes:
        - name: tmp-volume
          emptyDir: {}

score 0 · Answer 2 · answered Mar 22 '21 at 08:04

If you are testing it on your own kube setup, the metric server tries to connect on dns and as your setup does not have dns for nodes so you need to modify the configuration to allow the metric server to communicate on internal IP

add following flags to deployment

- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP

so overall deployment will look like this

apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      **k8s-app: metrics-server**
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      nodeSelector:
        kubernetes.io/os: linux
        kubernetes.io/arch: "amd64"

Thank you for your suggestion. I already set up as your writing. and the metrics server it self is working. like following commands. `kubectl top nodes/pods` — Atsushi Sakai, Mar 22 '21 at 08:12

score 0 · Answer 3 · answered Sep 03 '23 at 02:38

I often had to nudge kubernetes-dashboard to start working. In k8s 2.18 I just delete the two dashboard pods and when they are restarted dashboard magically starts working without that annoying 'Metric client health check failed' log messages. No configuration changes at all.

pzi@p31-ka11:~$ k get pod -n kubernetes-dashboard
NAME                                         READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-5657497c4c-45p5t   1/1     Running   0          151m
kubernetes-dashboard-78f87ddfc-bl9cg         1/1     Running   0          151m
pzi@p31-ka11:~$ k delete pod -n kubernetes-dashboard dashboard-metrics-scraper-5657497c4c-45p5t kubernetes-dashboard-78f87ddfc-bl9cg
pod "dashboard-metrics-scraper-5657497c4c-45p5t" deleted
pod "kubernetes-dashboard-78f87ddfc-bl9cg" deleted

k8s dashboard metric does not work (Metric client health check failed)

3 Answers3