17

Actual Issue:

Unable to start kubernetes API, due to which, unable to intite kube services like: kubectl version kubect get nodes

/home/ubuntu# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}



   The connection to the server localhost:8080 was refused - did you specify the right host or port  ?

Background of the issue:

Docker is installed. Using below, kube components are installed:

apt-get update && apt-get install -y kubeadm kubelet kubectl

But, when executing kubeadm init --apiserver-advertise-address=$myip --ignore-preflight-errors=all:

I0408 09:09:07.316109       1 client.go:352] scheme "" not registered, fallback to default scheme
I0408 09:09:07.319904       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0  <nil>}]
I0408 09:09:07.323010       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
W0408 09:09:07.332669       1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
I0408 09:09:08.325625       1 client.go:352] parsed scheme: ""
I0408 09:09:08.325650       1 client.go:352] scheme "" not registered, fallback to default scheme
I0408 09:09:08.325707       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0  <nil>}]
I0408 09:09:08.325768       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
W0408 09:09:08.326158       1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...

Getting the above in the kube api container logs. This is a fresh install .Also tried 

sudo cp /etc/kubernetes/admin.conf $HOME/
sudo chown $(id -u):$(id -g) $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf

Expected Results:

kubectl version: should give only the version details without any connection issue message Kubectl get nodes: should give the details of Master node and status

Suresh Vishnoi
  • 17,341
  • 8
  • 47
  • 55
Siva Malla
  • 255
  • 1
  • 2
  • 10
  • Can you try restarting docker? What port is your docker daemon running on? – Colwin Apr 08 '19 at 12:09
  • 2
    1) did you follow prerequisites checks from here - https://kubernetes.io/docs/setup/independent/#before-you-begin 2)Why do you use `--ignore-preflight-errors=all`? Can you try without it? 3) before moving further do `kubeadm reset` – A_Suh Apr 08 '19 at 12:37
  • If executed without the parameter, its not letting me go through: /home/ubuntu# kubeadm init --apiserver-advertise-address=xxxx [init] Using Kubernetes version: v1.14.0 [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` – Siva Malla Apr 08 '19 at 14:26
  • also tried with this parameter as all, but still seeing the same: somehow its trying to connect as : W0408 14:31:03.274065 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting... – Siva Malla Apr 08 '19 at 14:32
  • Its inability to connect to `etcd` (that's the `127.0.0.1:2379: connect: connection refused`) is very bad news, and that explains why you are getting connection refused by `kubectl` -- because the apiserver is not coming up – mdaniel Apr 09 '19 at 05:29
  • Can you please help in where and what needs to be changed in order to connect to the etcd – Siva Malla Apr 09 '19 at 16:41

3 Answers3

31

I spent a couple of hours on that.
In my case, I was using Ubuntu 22.04 and Kubernetes 1.24.

API was restarting all the time, I didn't find something in kubelet logs:

service kubelet status 
journalctl -xeu kubelet

I checked the API logs through:

/var/log/containers/kube-apiserver-XXXX

I saw the same error:

grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379

Solution:

containerd config default | tee /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml  
service containerd restart
service kubelet restart

Explanation:
The problem is cgroupv2 introduced with Ubuntu 21.04 and above(Debian since version 11).
You need to tell containerd to use the systemdCgroup driver via the config file(/etc/containerd/config.toml).
I'm using the default config file that containerd provides ( containerd config default | sudo tee /etc/containerd/config.toml).
Don't enable systemd_cgroup inside the "io.containerd.grpc.v1.cri" section, because the plugin doesn't seem support this anymore(status of the service will print following log:
"failed to load plugin io.containerd.grpc.v1.cri" error="invalid pluginconfig: systemd_cgroup only works for runtime io.containerd.runtime.v1.linux")
You need to enable the SystemdCgroup(=true) flag inside the section:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]"

Restart the containerd-service and then the kubelet-service or reboot your machine and then it should work as expected.

Credit: Dennis from slack

E235
  • 11,560
  • 24
  • 91
  • 141
  • 3
    I spent a week trying to find a solution to this problem....and finally I found your solution! It works! For others in similar situation, my setup is a 3 node Raspberry Pi cluster using Ubuntu 22.04 and kubeadm for the installation. Every time I ran "kubeadm init", the control plane would start up, but then after a while the kube-apiserver (:6443) could not connect to etcd (:2379). This made the kube-apiserver not work and create a lot of 6443 "dial tcp 192.168.0.105:6443: connect: connection refused" errors in the logs. – gummiost Sep 28 '22 at 13:22
  • 1
    Solution works perfectly fine with `RHEL9` and `kubelet` version `v1.25.3` – Vijay S B Nov 29 '22 at 10:12
  • Solution works with minor distro-specific changes for Fedora 36. (I took care to keep modify the defaults to preserve pre-existing paths in the old config) – Edwin Buck Dec 28 '22 at 16:47
  • I spent a 2 weeks starting over and over from fresh ubuntu 22 virtual machine install and kubeadm init, and every time my containers failed to start and k8s dont want to start. This solution helped me solve my problem. – Nessero Karuzo Jan 30 '23 at 09:04
1

The below has actually fixed my issue, not sure if this is the actual found:

apt-get update && \
 apt-get install -y apt-transport-https add-apt-repository "deb [arch=amd64] download.docker.com/linux/ubuntu bionic stable" curl -s packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - echo "deb apt.kubernetes.io kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list apt update && \
 apt install -qy docker.io apt-get update && \
 apt-get install -y kubeadm kubelet kubectl kubernetes-cni kubeadm init --ignore-preflight-errors=all –

I was installing docker.ce earlier as part of the kubernetes installation. Now I tried with docker.io which went good and no issues further. Its working as expected now.

tadman
  • 208,517
  • 23
  • 234
  • 262
Siva Malla
  • 255
  • 1
  • 2
  • 10
  • apt-get update && apt-get install -y apt-transport-https add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable" curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list apt update && apt install -qy docker.io apt-get update && apt-get install -y kubeadm kubelet kubectl kubernetes-cni kubeadm init --ignore-preflight-errors=all – Siva Malla Apr 10 '19 at 10:58
0

You're using cloud Instance, please use private IP of instance while running

kubeadm init
Peter Csala
  • 17,736
  • 16
  • 35
  • 75
santosh Kumar Verma
  • 1
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Sep 27 '21 at 09:38