0

Kubelet has been initialized with pod network for Calico :

sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --image-repository=someserver

Then i get calico.yaml v3.11 and applied it :

sudo kubectl --kubeconfig="/etc/kubernetes/admin.conf" apply -f calico.yaml

Right after i check on the pod status : 

sudo kubectl --kubeconfig="/etc/kubernetes/admin.conf" get nodes
NAME              STATUS     ROLES    AGE     VERSION
master-1   NotReady   master   7m21s   v1.17.2

on describe i've got cni config unitialized, but i thought that calico should have done that ?

MemoryPressure   False   Fri, 21 Feb 2020 10:14:24 +0100   Fri, 21 Feb 2020 10:09:00 +0100   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 21 Feb 2020 10:14:24 +0100   Fri, 21 Feb 2020 10:09:00 +0100   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 21 Feb 2020 10:14:24 +0100   Fri, 21 Feb 2020 10:09:00 +0100   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Fri, 21 Feb 2020 10:14:24 +0100   Fri, 21 Feb 2020 10:09:00 +0100   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

In fact i have nothing under /etc/cni/net.d/ so it seems it forgot something ?

ll /etc/cni/net.d/
total 0

sudo kubectl --kubeconfig="/etc/kubernetes/admin.conf" -n kube-system get pods
    NAME                                       READY   STATUS                  RESTARTS   AGE
calico-kube-controllers-5644fb7cf6-f7lqq   0/1     Pending                 0          3h
calico-node-f4xzh                          0/1     Init:ImagePullBackOff   0          3h
coredns-7fb8cdf968-bbqbz                   0/1     Pending                 0          3h24m
coredns-7fb8cdf968-vdnzx                   0/1     Pending                 0          3h24m
etcd-master-1                       1/1     Running                 0          3h24m
kube-apiserver-master-1            1/1     Running                 0          3h24m
kube-controller-manager-master-1    1/1     Running                 0          3h24m
kube-proxy-9m879                           1/1     Running                 0          3h24m
kube-scheduler-master-1             1/1     Running                 0          3h24m

As explained i'm running through a local repo and journalctl says :

 kubelet[21935]: E0225 14:30:54.830683   21935 pod_workers.go:191] Error syncing pod cec2f72b-844a-4d6b-8606-3aff06d4a36d ("calico-node-f4xzh_kube-system(cec2f72b-844a-4d6b-8606-3aff06d4a36d)"), skipping: failed to "StartContainer" for "upgrade-ipam" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://repo:10000/v2/calico/cni/manifests/v3.11.2: no basic auth credentials"
 kubelet[21935]: E0225 14:30:56.008989   21935 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Feels like it's not only CNI the issue

Albz
  • 41
  • 1
  • 5
  • what is the output of kubectl get pods -n kube-system? calico pods are running in kube-system namespace? – Arghya Sadhu Feb 21 '20 at 09:42
  • No it's not running, it says : calico-kube-controllers-5b644bc49c-w5cvq 0/1 Pending calico-node-fq22h 0/1 Init:ImagePullBackOff Cordns are also pending – Albz Feb 21 '20 at 10:04
  • Can you kubectl describe and kubectl logs on calico controller and calico node pods? Do you have access to download calico images from outside docker repo? What is that --image-repository=someserver? – Arghya Sadhu Feb 21 '20 at 10:08
  • --image-repository=someserver is my repo server in internal, we've got proxies and firewall so we manage to use an internal repo, this works for deploying docker images. describe calico pod says : - Pulling image "calico/cni:v3.11.2" - Failed to pull image "calico/cni:v3.11.2": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) - Back-off pulling image "calico/cni:v3.11.2" - Error: ErrImagePull - Error: ImagePullBackOff – Albz Feb 21 '20 at 10:22
  • @Albz yes it seems the main problem is really that it's not being able to pull `calico/cni:v3.11.2` image. Add it to your local repo and try again, if you still face problems let me know! – Will R.O.F. Feb 21 '20 at 10:55
  • add output of `kubectl get pods --all-namespaces` to your question – morgwai Feb 21 '20 at 21:28
  • @morgwai : NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-5644fb7cf6-f7lqq 0/1 Pending 0 29m kube-system calico-node-f4xzh 0/1 Init:ImagePullBackOff 0 29m kube-system coredns-7fb8cdf968-bbqbz 0/1 Pending 0 53m kube-system coredns-7fb8cdf968-vdnzx 0/1 Pending 0 53m ... – Albz Feb 25 '20 at 10:59
  • @Albz in the question, man... geez... how do you expect anyone to read what you just provided? – morgwai Feb 25 '20 at 12:03
  • Yes that's done – Albz Feb 25 '20 at 14:46

2 Answers2

2

Core DNS pod will be pending and master will be NotReady till calico pods are successfully running and CNI is setup properly.

It seems to be network issue to download calico docker images from docker.io. So you can pull calico images from docker.io and and push it to your internal container registry and then modify the calico yaml to refer that registry in images section of calico.yaml and finally apply the modified calico yaml to the kubernetes cluster.

Arghya Sadhu
  • 41,002
  • 9
  • 78
  • 107
  • Ok i manage to deploy calico node from my private repo. But still it is stuck with strange errors : pod_workers.go:191] Error syncing pod cec2f72b-844a-4d6b-8606-3aff06d4a36d ("calico-node-f4xzh_kube-system(cec2f72b-844a-4d6b-8606-3aff06d4a36d)"), skipping: failed to "StartContainer" for "upgrade-ipam" with ImagePullBackOff: "Back-off pulling image \"private-repo:10000/calico/cni:v3.11.2\"" And stays in Init:ErrImagePull – Albz Feb 25 '20 at 10:56
  • Did you **really** added the calico/cni:3.11.2 on your private repo? because the error seems to be the same but now it's saying it can't pull the image from `"private-repo:10000/calico/cni:v3.11.2\"` Are you using Proxy in your network?? – Will R.O.F. Feb 25 '20 at 12:59
2

So the issue with Init:ImagePullBackOff was that it cannot apply image from my private repo automatically. I had to pull all images for calico from docker. Then i deleted the calico pod it's recreate itself with the newly pushed image 

sudo docker pull private-repo/calico/pod2daemon-flexvol:v3.11.2
sudo docker pull private-repo/calico/node:v3.11.2
sudo docker pull private-repo/calico/cni:v3.11.2
sudo docker pull private-repo/calico/kube-controllers:v3.11.2

sudo kubectl -n kube-system delete po/calico-node-y7g5

After that the node re-do all the init phase and :

sudo kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-5644fb7cf6-qkf47   1/1     Running   0          11s
calico-node-mkcsr                          1/1     Running   0          21m
coredns-7fb8cdf968-bgqvj                   1/1     Running   0          37m
coredns-7fb8cdf968-v85jx                   1/1     Running   0          37m
etcd-lin-1k8w1dv-vmh                       1/1     Running   0          38m
kube-apiserver-lin-1k8w1dv-vmh             1/1     Running   0          38m
kube-controller-manager-lin-1k8w1dv-vmh    1/1     Running   0          38m
kube-proxy-9hkns                           1/1     Running   0          37m
kube-scheduler-lin-1k8w1dv-vmh             1/1     Running   0          38m
Albz
  • 41
  • 1
  • 5