0

I am diving into K8s and deployed my first pod via kubectl apply on the master node. In order to check its status, I called kubectl get pods twice in a row. I did not touch anything, but a subsequent call of the same command failed with the error below. Can anyone help me understand what may have happened?

ubuntu:~$ kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   0/1     Pending   0          18s
ubuntu:~$ kubectl get pods
The connection to the server xxx.xxx.xxx.xxx:6443 was refused - did you specify the right host or port?

For completion, here the status of kubelet.service on the master node:

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Thu 2023-04-20 02:12:52 UTC; 23min ago

Thanks for any hint.

Daniel Stephens
  • 2,371
  • 8
  • 34
  • 86
  • Most probably it is your config on your machine. Check your config `kubectl config view`. Also check https://stackoverflow.com/a/37188830/2659499 – Vahid Apr 20 '23 at 02:39
  • I will post that, but would that explain the first command going through? Any service that I can check that may have stopped? – Daniel Stephens Apr 20 '23 at 02:40
  • That error means literally that: the connection to the kubernetes api server was refused. Contact the server administrator and report they are having problems. – zerkms Apr 20 '23 at 02:42
  • Did you change context by any chance before running for the second time? – Vahid Apr 20 '23 at 02:43
  • Does it continue to fail after that, or is it intermittent? Are you able to get anything useful from `/var/log/{containers,kubernetes}`? – Zac Anger Apr 20 '23 at 02:43
  • @zerkms I understand, my question was what could cause this without doing anything except calling the same command twice – Daniel Stephens Apr 20 '23 at 02:47
  • @Vahid It keeps failing, and I don't know which service to restart – Daniel Stephens Apr 20 '23 at 02:47
  • Check the health of your API server `kubectl get --raw /healthz` [Kubernetes API health endpoints](https://kubernetes.io/docs/reference/using-api/health-checks/) – Vahid Apr 20 '23 at 02:54
  • Same, the connection got refused :-/ It seems the entire service is down, despite `kubelet.service` reports to be active (not sure if there is another service missing) – Daniel Stephens Apr 20 '23 at 02:56
  • 1
    Kubelet is a separate component than API server. Why are you checking that? – Vahid Apr 20 '23 at 02:57
  • 1
    On the master, you could do `netstat -a | grep 6443` (or `lsof -i :6443`) to see if kube server is listening on the right port still (and of course using `ps`, `htop`, whatever, to see if it's even running). Try also checking logs at /var/log if you have access to them. Also, kubelet is on nodes, kube-apiserver is how kubectl and various other tools interact with your cluster. – Zac Anger Apr 20 '23 at 02:57
  • 1
    check the log `journalctl -u kube-apiserver -n 100` and the status `systemctl status kube-apiserver` – Vahid Apr 20 '23 at 03:05
  • This is so bizarr, `Unit kube-apiserver.service could not be found.` Seems somethingis completely broken on my setup. I have now a different entry point to fix – Daniel Stephens Apr 20 '23 at 03:07
  • 1
    There are some other questions on StackOverflow and ServerFault about that, and some related Github issues. But it sounds like the service file is malformed, in the wrong place, has the wrong permissions, or (lots of other potential problems, but a SystemD issue in general) — or you don't have one to begin with. – Zac Anger Apr 20 '23 at 03:13
  • 1
    Also, just a comment not directly related to your problem, but something to keep in mind — running K8s from scratch is notoriously painful; you're just at the start of it and already facing issues, and that's why there are so many platforms built on top of K8s to make it easier. But it is a good learning experience. – Zac Anger Apr 20 '23 at 03:15
  • Thanks for the help. Yes, K8s is by far the most painful experience at start I had in a long time. – Daniel Stephens Apr 20 '23 at 03:42

1 Answers1

1

As explained in this doc by Greek Diary ‘admin’ which explains how to fix the kubectl error:The connection to the server x.x.x.x:6443 was refused - did you specify the right host or port?

Below troubleshooting steps will help to resolve your issue:

  1. The kubectl should be executed on the Master Node.

  2. Current user must have Kubernetes cluster configuration environment variable (Details of how to are listed under section Preparing to Use Kubernetes as a Regular User),

    $ env | grep -i kube

    KUBECONFIG=/root/.kube/config

3.The docker service must be running:

 $ systemctl status docker

4.The kubelet service must be running:

   $ systemctl status kubelet

5.TCP port 6443 should be listed as it is listening to port:

netstat -pnlt | grep 6443

If TCP port 6443 is not available, check firewall/iptables rules matching requirements:

$ firewall-cmd --list-all

Also check kubelet logs:

# journalctl -xeu kubelet

6.Try restarting Kubernetes cluster which will also do some basic checks:

   $ kubeadm-setup.sh restart
Sai Chandini Routhu
  • 750
  • 1
  • 3
  • 13