dnsutils pod giving "incompatible CNI versions" error in kubernetes

Question

I made a kubernetes cluster. All the default pods seem to have initialized fine but when I try to add a dnsutils pod I get the following error:

Events:
  Type     Reason                  Age               From               Message
  ----     ------                  ----              ----               -------
  Normal   Scheduled               3m29s             default-scheduler  Successfully assigned default/dnsutils to node-4
  Warning  FailedCreatePodSandBox  3m29s             kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0ee2103d7d204484a39d5a9ad6474704ddc0bf1f1557238b754c89c2d071b00e": plugin type="bridge" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
  Warning  FailedCreatePodSandBox  3m15s             kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3b46b71c5a4c924f13f8a344940a883cb91247dfca95668384d763b403e02d42": plugin type="bridge" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
  Warning  FailedCreatePodSandBox  3m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a2b7b5a0c558fad41e6b65246b41ff8959cbf0259a94299abe496c983a77c27a": plugin type="bridge" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
  Warning  FailedCreatePodSandBox  2m47s             kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2ee767296a625c213f6f09f24a37bb73bb673ee60c2d2df13163c865f1248074": plugin type="bridge" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
  Warning  FailedCreatePodSandBox  2m32s             kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d29b1497142a7abc46e714001f183bed8e68798a5b5f2b559afd9e2eff17c377": plugin type="bridge" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
  Warning  FailedCreatePodSandBox  2m17s             kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "862a82892c68397ae7df7d0c410bda7b46522c79c1b9eb61a1ef575d4d09e0a9": plugin type="bridge" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
  Warning  FailedCreatePodSandBox  2m2s              kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "03000265ca15443c0b571152f97d5e4f95f67437dfe72bc0cd18bc9132f181cd": plugin type="bridge" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
  Warning  FailedCreatePodSandBox  107s              kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5a02ac5807e58b6aa7c0fac1d807533ff61f3bd223600e5d5177f96a22f9253b": plugin type="bridge" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
  Warning  FailedCreatePodSandBox  95s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7d15145fc5d4041d923824915f5f0bb0090e8e9e4e86cd91a2fa94b11b69e18c": plugin type="bridge" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
  Warning  FailedCreatePodSandBox  6s (x7 over 84s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4d72686240042e4764d3475e8a2aeeeef08f2b20025e58ca4e1860c628ca42bc": plugin type="bridge" failed (add): incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]

I'm using flannel as my CNI plugin. I got this error before on my coredns pods too when I was using weave net. The Kubernetes website (https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) says they only support v0.4.0 of the CNI specification, so I'm not sure why it's trying to use v1.0.0 or how I would go about changing that.

Which version of kubernetes do you use? Could you share your yaml configuration with dnsutil pod? — RadekW, Mar 22 '22 at 10:59
I'm using version 1.23.5. The dnsustils pod is taken from the kubernetes site: https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ — Matthew Tromp, Mar 22 '22 at 15:52
I deployed this dnsutils pod from link you sent and it worked for me. I also use flannel CNI in my cluster. Did you change something in your CNI configuration or in this dnsutils yaml? When did you deploy flannel CNI to your cluster? Could you share yaml of this flannel? I would like to compare with mine — RadekW, Mar 23 '22 at 11:27
I didn't change anything in any of my configurations. They're all just right out of the box. I deployed flannel to my cluster right before trying to deploy the dnsutils container. Here's the yaml: https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml — Matthew Tromp, Mar 25 '22 at 14:54
My CNI config is a little different. I have lines in my CNI: `image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.0` and your is set to `v1.0.1` and also `image: quay.io/coreos/flannel:v0.15.1` which your is set to `v0.17.0`. Could you change is to older version and try? — RadekW, Mar 28 '22 at 09:31
There's no `quay.io/coreos/flannel` line in my yaml. I have `image: rancher/mirrored-flannelcni-flannel:v0.17.0` but if I try to change that to `v0.15.1` it gives an imagepullback error — Matthew Tromp, Mar 28 '22 at 14:40
Alright nevermind. I managed to fix it, not sure how. It involved removing the worker nodes from the cluster, changing their container runtime (the master was using cri-o while the rest were using containerd because I did something wrong) and readding them. No idea what I did wrong but it's fixed now — Matthew Tromp, Mar 28 '22 at 17:01
Glad to hear that you found a solution. Could you post an answer with explanation? As you can read [here](https://stackoverflow.com/help/self-answer) it is very good practice and it will be helpful in future for other people — RadekW, Mar 29 '22 at 09:11

score 1 · Answer 1 · answered Mar 29 '22 at 12:16

So I fixed it, somehow. My master node was running CRI-O while my workers were using containerd. To fix this I had to uninstall containerd from the workers, install CRI-O and podman onto all of them, and then I had a separate issue where kubelet kept trying to use docker even though I uninstalled docker, which I fixed by uninstalling kubelet, removing all of its related files with rm -rf (find / *kubelet*) (probably not the safest way of doing that) and reinstalling it. Then when I used kubeadm to remake the cluster everything worked for some reason

dnsutils pod giving "incompatible CNI versions" error in kubernetes

1 Answers1