1

In some cases after a node reboot the Kubernetes cluster managed by MicroK8S cannot start pods.

With a describe of the pod failing to be ready I could see that the pod was stuck in a "Pulling image" state during several minutes without any other event, as following:

Events:
  Type     Reason     Age                 From                     Message
  ----     ------     ----                ----                     -------
  Normal   Scheduled  50s                 default-scheduler        Successfully assigned default/my-service-abc to my-desktop
  Normal   Pulling    8m                  kubelet                  Pulling image "127.0.0.1:32000/some-service"

Pulling from the node with docker pull 127.0.0.1:32000/some-service works perfectly, so it doesn't seem a problem with Docker.

I have upgraded Docker just in case.

I seem to run the latest version of microk8s.

David Maze
  • 130,717
  • 29
  • 175
  • 215
Micaël Félix
  • 2,697
  • 5
  • 34
  • 46

1 Answers1

2

Running a sudo microk8s inspect gives no error/warning, everything seemed to work perfectly.

As a Docker pull did work locally, it's really the Kubelet app which is communicating with Docker that seemed stuck.

Even with a sudo service docker stop && sudo service docker start it did not work.

Even a rollout did not suffice to get out of the Pulling state after the reboot of docker.

Worst of all, a reboot of the server did not change anything (the pod that were up were still working, but all the other pods (70%) were down and in ContainerCreating state.

Checking the status systemctl status snap.microk8s.daemon-kubelet did not report any error.

The only thing that worked seemed to do this:

sudo systemctl reboot snap.microk8s.daemon-kubelet

However, it also rebooted the whole node, so that's something to do in a last case scenario (same as the node reboot).

Micaël Félix
  • 2,697
  • 5
  • 34
  • 46