7

I have an image of size of 6.5GB in the Google Container Registry. When I try to pull the image on a Kubernetes cluster node(worker node) via a deployment, an error occurs: ErrImagePull(or sometimes ImagePullBackOff). I used the describe command to see the error in detail. The error is described as Failed to pull image "gcr.io/.../.. ": rpc error: code = Canceled desc = context canceled What may be the issue and how to mitigate it?

Ajay
  • 75
  • 1
  • 1
  • 6

2 Answers2

7

It seems that the kubelet expects a updates on progress during the pull of a large image but this currently isn't available by default with most container registries. It's not ideal behaviour but it appears people have been able to work around it from reading the responses on https://github.com/kubernetes/kubernetes/issues/59376 and Kubernetes set a timeout limit on image pulls by adjusting the timeout

Ryan Dawson
  • 11,832
  • 5
  • 38
  • 61
  • The first link provides solutions where the --image-pull-progress-deadline flag is set to a higher value. I tried setting this in the worker node. But the error is: **error reading /var/lib/kubelet/pki/kubelet.key, certificate and key must be supplied as a pair**. The second link that you have provided talks about the --runtime-request-timeout flag. This flag has now been deprecated. – Ajay Aug 16 '18 at 07:09
  • I guess you set in the KubeletArguments of node-config.yaml like in https://github.com/openshift/origin/issues/13122#issuecomment-330248646 ? What value did you set it to? I wonder if that error could mean the value is invalid – Ryan Dawson Aug 16 '18 at 07:21
  • 1
    Runtime-request-timeout is still in the official docs https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ Do you have a link on it being deprecated? – Ryan Dawson Aug 16 '18 at 07:25
  • Yes, I saw that the runtime-request-timeout flag is still in the official docs. But when I use it on the command-line, the output tells me that the flag is deprecated and that I should set it in the Kubelet config file. – Ajay Aug 16 '18 at 07:39
  • I would imagine you can use the deprecated approach for now as a test and then move to the file-based approach (https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/, https://github.com/kubernetes/kubernetes/issues/62023, https://github.com/kubernetes/kubernetes/issues/34915) – Ryan Dawson Aug 16 '18 at 08:03
  • Actually it might be --image-pull-progress-deadline you want. Another way might be to go on to the node and `docker pull` the image so that it becomes available locally. See https://groups.google.com/forum/m/#!topic/kubernetes-sig-storage-bugs/3KHBY7zF6z0 – Ryan Dawson Aug 16 '18 at 08:34
  • 2
    Thank you for all the inputs. I have tried --image-pull-progress-deadline. I set it to 60m. But I wasn't sure if it was configured, and if it was configured, how to restart kubelet. I edited the runtimeRequestTimeout field in the config file(in /var/lib/kubelet) and then pulled a relatively smaller 1.5GB image(This had also failed previously). It was pulled successfully. – Ajay Aug 16 '18 at 10:09
  • Did this kubelet flag get deprecated? I can't find it in the [current kubelet docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) – j7skov Aug 10 '23 at 17:10
1

Use --image-pull-progress-deadline duration as a parameter when you start the kubelet.

This is documented in the kubelet documentation.

If no pulling progress is made before this deadline, the image pulling will be cancelled. (default 1m0s)

lanoxx
  • 12,249
  • 13
  • 87
  • 142
  • 1
    Just a note that his will be deprecated along with dockershim in Kubernetes 1.24 - https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/ – nedstark179 Apr 19 '22 at 11:18
  • @nedstark179 Is there a better solution with `kubelet config set [] []`? – Manouchehri Jul 18 '22 at 17:23