0

I'm using Rancher 2.5.8 to manage my Kubernetes clusters. Today, I created a new cluster and everything worked as expected, except the metrics-server. The status of the metrics-server is always "CrashLoopBackOff" and the logs are telling me the following:

E0519 11:46:39.225804       1 server.go:132] unable to fully scrape metrics: [unable to fully scrape metrics from node worker1: unable to fetch metrics from node worker1: unable to extract connection information for node "worker1": node worker1 had no addresses that matched types [InternalIP], unable to fully scrape metrics from node worker2: unable to fetch metrics from node worker2: unable to extract connection information for node "worker2": node worker2 had no addresses that matched types [InternalIP], unable to fully scrape metrics from node worker3: unable to fetch metrics from node worker3: unable to extract connection information for node "worker3": node worker3 had no addresses that matched types [InternalIP], unable to fully scrape metrics from node main1: unable to fetch metrics from node main1: unable to extract connection information for node "main1": node main1 had no addresses that matched types [InternalIP]]
I0519 11:46:39.228205       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0519 11:46:39.228222       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0519 11:46:39.228290       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0519 11:46:39.228301       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0519 11:46:39.228310       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0519 11:46:39.228314       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0519 11:46:39.229241       1 secure_serving.go:197] Serving securely on [::]:4443
I0519 11:46:39.229280       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I0519 11:46:39.229302       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0519 11:46:39.328399       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I0519 11:46:39.328428       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0519 11:46:39.328505       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController

Is anyone having any idea how I can solve the issue so that the metrics-server isn't crashing anymore?

Here's the output of the kubectl get nodes worker1 -oyaml:

status:
  addresses:
  - address: worker1
    type: Hostname
  - address: 65.21.<any>.<ip>
    type: ExternalIP
Nrgyzer
  • 783
  • 1
  • 14
  • 38
  • Can you run `kubectl get nodes worker1 -oyaml` and add `.status.addresses` list to your question? What types of addresses do you see (there is a `type:` field)? – Matt May 20 '21 at 08:26
  • I've added the status-part of the worker1. Any idea? – Nrgyzer May 20 '21 at 12:39
  • Since there is no InternalIP address, you can try to use the one that there is. Your metrics-server most probably has this line: `--kubelet-preferred-address-types=InternalIP` in deployment. Change it's value to `Hostname` or `ExternalIP`. Athough I dont know why you dont have the InternalIP type address. Where do you run your cluster? – Matt May 20 '21 at 14:01
  • Yeah, I changed the preferred address to ExternalIP and it works. My cluster is running on Hetzner servers and I configured the Hetzner Cloud Controller addon when setting up the cluster. – Nrgyzer May 20 '21 at 17:52
  • Great. Would you mind to provide the answer, describing exactly what you did so that it can help others in future? I'd be happy to upvote. – Matt May 21 '21 at 05:43

1 Answers1

0

The issue was with the metrics server.

Metrics server was configured to use kubelet-preferred-address-types=InternalIP but worker node didn't have any InternalIP listed:

$ kubectl get nodes worker1 -oyaml:
[...]
status:
  addresses:
  - address: worker1
    type: Hostname
  - address: 65.21.<any>.<ip>
    type: ExternalIP

The solution was to set --kubelet-preferred-address-types=ExternalIP in metrics server deployment yaml.

But probably better solution would be to configure it as in official metrics server deployment yaml (source):

- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

As states in metrics-server configuration docs:

--kubelet-preferred-address-types - The priority of node address types used when determining an address for connecting to a particular node (default [Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP])

Matt
  • 7,419
  • 1
  • 11
  • 22