1

Am facing an unexpected issue today, earlier today I noticed that "kubectl get pods" was returning an "Unable to connect to the server: EOF". Upon further investigation I found out that Kubernetes apiserver is unable to connect to 127.0.0.1:443. I have been unable to resolve this problem, any assistance would be highly appreciated. Below are the logs I found.

  Nov 20 18:13:30 ip-172-31-152-166.us-west-2.compute.internal kubelet[6398]: E1120 18:13:30.362106    6398 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:481: Failed to list *v1.Node: Get https://127.0.0.1/api/v1/nodes?fieldSelector=metadata.name%3Dip-172-31-152-166.us-west-2.compute.internal&limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: getsockopt: connection refused
    Nov 20 18:13:30 ip-172-31-152-166.us-west-2.compute.internal kubelet[6398]: E1120 18:13:30.362928    6398 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://127.0.0.1/api/v1/pods?fieldSelector=spec.nodeName%3Dip-172-31-152-166.us-west-2.compute.internal&limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: getsockopt: connection refused
    Nov 20 18:13:30 ip-172-31-152-166.us-west-2.compute.internal kubelet[6398]: E1120 18:13:30.363719    6398 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:472: Failed to list *v1.Service: Get https://127.0.0.1/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: getsockopt: connection refused
Collins C.
  • 151
  • 1
  • 4
  • Hi, what platform/infrastructure do you have? Can you ssh to master and check if the containers are running with `docker ps`? What is your kubernetes version? – Piotr Malec Nov 21 '19 at 15:31
  • Hello we are running a wso2 docker environment, yes I can ssh to the master and all the containers are running except for api-server manager which has the error shown above. We are using Kubernetes v1.13.3. – Collins C. Nov 26 '19 at 07:55
  • Check if You ran out of free disk space. Also verify that swap is disabled. Was the node host rebooted or powered-off? – Piotr Malec Nov 26 '19 at 12:18
  • There is available free space, also swap is disabled and the node host was not rebooted or powered off. – Collins C. Nov 26 '19 at 15:18
  • Check logs in /var/log/kube-apiserver.log . In case of systemd-based systems, you may need to use journalctl instead. If you want You can also try to restart kubelet and docker services. – Piotr Malec Nov 27 '19 at 14:00
  • Thank you Piotr, however I had to recreate our environment from scratch as there was no working solution. Thank you for your advice. – Collins C. Dec 10 '19 at 12:52

0 Answers0