0
  • k8s cannot successfully create pod and is always in ConainerCreating stage.
  • When the describe pod status reports the error message:
    • network: failed to find plugin bandwidth in path [/opt/cni/bin]]
    • Pod sandbox changed, it will be killer and re-create.
Events:
  Type     Reason                  Age               From               Message
  ----     ------                  ----              ----               -------
  Normal   Scheduled               22s               default-scheduler  Successfully assigned monitoring/vpc-minibase-fronted-monitor-687545864b-2tfr7 to 192.168.1.90
  Warning  FailedCreatePodSandBox  22s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "df25ea8a9310e4bac83aa7661e8b0304991790c4bf727ff7272738a460a229f2" network for pod "vpc-minibase-fronted-monitor-687545864b-2tfr7": networkPlugin cni failed to set up pod "vpc-minibase-fronted-monitor-687545864b-2tfr7_monitoring" network: failed to find plugin "loopback" in path [/opt/cni/bin], failed to clean up sandbox container "df25ea8a9310e4bac83aa7661e8b0304991790c4bf727ff7272738a460a229f2" network for pod "vpc-minibase-fronted-monitor-687545864b-2tfr7": networkPlugin cni failed to teardown pod "vpc-minibase-fronted-monitor-687545864b-2tfr7_monitoring" network: failed to find plugin "bandwidth" in path [/opt/cni/bin]]
  Normal   SandboxChanged          8s (x2 over 21s)  kubelet            Pod sandbox changed, it will be killed and re-created.
  • Check the cni path /opt/cni/bin/bandwidth but it do exists.
> ls /opt/cni/bin/

bandwidth  calico  calico-ipam  flannel  host-local  install  loopback  portmap  tuning
niewenhan
  • 1
  • 2

1 Answers1

0
  1. Since the path /opt/cni/bin/bandwidth exists, it is suspected that the cni plugin cannot be correctly identified because of the inode change.
  2. Do the experiment and change the file inode in turn.
  • Change the inode of /opt/cni/bin/bandwidth, but pod can still created.
  • Change the inode of /opt/cni/bin, but pod can still created.
  • Change the inode of /opt/cni, and reproduce the above phenomenon.
> sudo cp cni cni.bak
> ls -li

2737603 drwxr-xr-x  3 root     root     4096 Sep 22 20:56 cni
2737669 drwxr-xr-x  3 root     root     4096 Sep 22 20:58 cni.bak

> sudo rm -rf cni
> sudo mv cni.bak cni

> ls -li

2737669 drwxr-xr-x  3 root     root     4096 Sep 22 20:56 cni
  1. Try restarting kubelet to get the current environment cni plugin again, since then all pods are back to normal.
docker restart kubelet
niewenhan
  • 1
  • 2