1

We have a statefulset that we want to have minimum downtime (like any other statefulset out there I suppose), but the pod gets stuck at "terminating" state since the readiness probe failure threshold is 5 (adds a 5s downtime during "Terminating" state for no reason). So, for it to terminate faster, I reduced the failure threshold to 1, it now terminates faster, however now that the failure threshold is so low, it causes random unready pods once in a while due to the CPU hitting 100% or other transient problems.

My question: How can I make the pods terminate faster while keeping Readiness failure threshold a high number to reduce the downtime during pod restart? (Also, I would appreciate any other random tips that would make the pods restart faster (e.g. spend less time in ContainerCreating)

hoque
  • 5,735
  • 1
  • 19
  • 29
Oguz Yildiz
  • 51
  • 1
  • 7
  • What is your environment (Kubectl version, using kubeadm/minikube). Its our local env or your cluster is in Cloud? Whould it be possible to provide your configuration YAMLs? – PjoterS May 19 '20 at 12:30
  • Sorry @PjoterS , I'm just seeing your reply now. We're using Amazon EKS 1.16. What parts of the yaml would you like to see? – Oguz Yildiz Jun 23 '20 at 20:03

1 Answers1

0

If you want to delete a Pod forcibly using kubectl version >= 1.5, do the following:

kubectl delete pods <pod> --grace-period=0 --force

If you’re using any version of kubectl <= 1.4, you should omit the --force option and use:

kubectl delete pods <pod> --grace-period=0

If even after these commands the pod is stuck on Unknown state, use the following command to remove the pod from the cluster:

kubectl patch pod <pod> -p '{"metadata":{"finalizers":null}}'

Ref: https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/

hoque
  • 5,735
  • 1
  • 19
  • 29
  • Hi, thanks for your response! However this doesn't solve my problem, I'm not talking about manually deleting the pod, I'm talking about the rolling deployment situation. In my case, I need the graceful termination because the pod is saving some state to a storage. but after it's done and the main container terminates, the pod wastes another 5s in there because the readiness doesn't fail. But once the container is terminated, I want the pod gone immediately so that it can restart sooner – Oguz Yildiz May 18 '20 at 23:44