4

Environment:

  • Kubernetes 1.12.2
  • Docker 18.9.0
  • microk8s.kubectl

$ k get all

NAME                                          READY   STATUS             
RESTARTS   AGE
pod/mysql-0                                   1/1     Running            0          72s
pod/nginx-ingress-microk8s-controller-c2pgz   0/1     CrashLoopBackOff   129        22h
pod/web-0                                     1/1     Running            0          78s

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/kubernetes      ClusterIP   10.152.183.1   <none>        443/TCP    70m
service/mysql-service   ClusterIP   None           <none>        3306/TCP   72s
service/nginx-service   ClusterIP   None           <none>        80/TCP     78s

NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress-microk8s-controller   1         1         0       1            0           <none>          2d22h

NAME                     DESIRED   CURRENT   AGE
statefulset.apps/mysql   1         1         72s
statefulset.apps/web     1         1         78s

/var/log/syslog:

failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory

Error syncing pod f0ab0f74-e6f2-11e8-8410-482ae31e6a94 ("nginx-ingress-microk8s-controller-c2pgz_default(f0ab0f74-e6f2-11e8-8410-482ae31e6a94)"), skipping: failed to "StartContainer" for "nginx-ingress-microk8s" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=nginx-ingress-microk8s pod=nginx-ingress-microk8s-controller-c2pgz_default(f0ab0f74-e6f2-11e8-8410-482ae31e6a94)"

What is nginx-ingress-microk8s-controller-c2pgz? Who started it?

Kok How Teh
  • 3,298
  • 6
  • 47
  • 85
  • Is it happening while joining the worker node? Or the system was already running and it started happening? Is /var/lib/kubelet/config.yaml present on the system? – Prafull Ladha Nov 13 '18 at 09:03
  • No. The file is missing. – Kok How Teh Nov 13 '18 at 13:16
  • what do you mean by "No"? Can you share information what you are trying to do, your cluster architecture and when exactly you faces with this error? – Vit Nov 13 '18 at 15:24
  • I think it is due to the fact that kubeadm init fails: https://github.com/kubernetes/kubeadm/issues/1239 – Kok How Teh Nov 14 '18 at 06:02

3 Answers3

3

You mentioned in the comments that the reason is related to kubeadm init fails.

The /var/lib/kubelet/config.yaml config file is being populated only after:

  1. A successful cluster initialization (kubeadmin init) in the master node.

  2. In the worker node - after a successful joining to the cluster (kubeadm join).

So if the problem is with kubeadm init you should check the command's output (also great if you could paste it in the question).
Make sure you don't run kubeadm init with the --ignore-preflight-errors=all flag.

I'm not familiar with your specific error, but in order for the answer to be more helpful - I'll try to give some possible solutions:

  1. Make sure all requirements for kubeadm are in place.

  2. Check the firewall rules - make sure you don't block egress traffic and that port 6443 ingress rule is open for the worker node (relevant for the joining phase).

  3. Make sure that the required ports are not occupied.

  4. Try restarting Kubelet with systemctl restart kubelet and check latest logs with:
    sudo journalctl -u kubelet -n 100 --no-pager.

  5. Check if Docker version can be updated to a newer stabler one.

  6. Try running kubeadm reset and make sure you re-run kubeadm init with latest version or with the specific stable version by addding --kubernetes-version=X.Y.Z.

Rot-man
  • 18,045
  • 12
  • 118
  • 124
0

As per RtmY, it works only kubectl initilzation works correct after doing following

kubeadm  init --pod-network-cidr=192.168.0.0/16

it worked successfully.

Veerendra K
  • 2,145
  • 7
  • 32
  • 61
Bhagyaraj
  • 1
-2

As i have updated kubelet, I am not able to find /var/lib/kubelet/config.yaml

For that "systemctl status kubelet|journalctl -xe"

failed to load Kubelet config file /var/lib/kubelet/config.yaml

As per the below link, I have copied the config.yaml from other working worker nodes and its worked !!

https://github.com/kubernetes/kubernetes/issues/65863#issuecomment-403003592

amit23comp
  • 1,483
  • 1
  • 13
  • 12
  • 1
    A link to a solution is welcome, but please ensure your answer is useful without it: [add context around the link](//meta.stackexchange.com/a/8259) so your fellow users will have some idea what it is and why it’s there, then quote the most relevant part of the page you're linking to in case the target page is unavailable. [Answers that are little more than a link may be deleted.](//stackoverflow.com/help/deleted-answers) – double-beep Aug 13 '19 at 10:49
  • This is not working. – Stephan Kristyn Jan 09 '23 at 20:12