0

I have a Kubernetes cluster, and I am studying the behaviour of Kubernetes pod in case of failure.

So in my single master 2 worker node Kubernetes cluster, that has one StatefulSet deployed. If I inject a failure(intentionally) by changing the image of the pod in the StatefulSet spec to a non-existent version and considering the strategy Type is RollingUpdate, I see pod keeps trying for hours/days to create a container and doesn't stop this effort. Is there a configuration in Kubernetes that will let K8s take an action in such scenario?

sample-0   1/1    Running            0         23h  worker1
sample-1   0/1    ImagePullBackOff   0         23h  worker2
Shresthi Garg
  • 15
  • 4

1 Answers1

1

The back-off is unfortunately not tunable.

From Kubernetes official documentation:

The BackOff part indicates that Kubernetes will keep trying to pull the image, with an increasing back-off delay.

Kubernetes raises the delay between each attempt until it reaches a compiled-in limit, which is 300 seconds (5 minutes).

Reference: https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy

CaioT
  • 1,973
  • 1
  • 11
  • 20
  • Thanks for taking out time to answer this. So, I did read this, but didn't understand the entire meaning. Does that mean only the delay been the trial can be a maximum of 300 sec but the number of back-off trials is not mentioned, so thus it can go on forever (until of course manually managed)? Correct me if I am wrong. – Shresthi Garg Jul 22 '21 at 21:30
  • No, the backoff delay is increased after each attempt but capped at 5 minutes (300 seconds). Maybe this will give you some better ideia - https://github.com/kubernetes/kubernetes/issues/57291 – CaioT Jul 23 '21 at 12:03