0

Consider I have 10 containers in my POD.

I have added startUpProbe for 3 containers.

If I delete my POD, before Probe is completed sucessfully (Which means those containers are not in READY state)

Deleting Pod should send SIGTERM signal to all the containers to down it gracefully.

Will SIGTERM send to 3 containers where am using probes that are not yet completed ?

David Maze
  • 130,717
  • 29
  • 175
  • 215
sethu ram
  • 23
  • 1
  • 6
  • Can you remove the ambiguity here by breaking this into 10 separate Deployments with one container each? You should be able to simulate this pretty easily by launching a two-container Deployment where one never answers its health check (maybe it has an HTTP probe but `command: [sleep, infinity]`) and the other does something visible on SIGTERM; what happens there? – David Maze May 23 '23 at 10:49
  • May be for future. but currently these 10 containers are has to place in a single pod. – sethu ram May 24 '23 at 03:47

1 Answers1

1

The blog Troubleshooting SIGTERM: graceful termination of Linux containers written by James Walker details about SIGTERM and how to leverage it in your application so that it can terminate without corrupting any data.

SIGTERM is a Linux signal that Unix-based operating systems issue when they want to terminate a running process. In normal circumstances, your application should respond to a SIGTERM by running cleanup procedures that facilitate graceful termination. If processes aren’t ready to terminate, they may elect to ignore or block the signal.

So, when a pod is deleted the SIGTERM signal is sent to all the containers irrespective of their state and the pod will wait till the graceful termination period(which is 30s by default), once the graceful period ends SIGKILL signal will be sent which will forcefully terminates all the processes. Since your containers are having startUpProbe enabled they will be in the pending state when the SIGTERM signal is generated and can continue the process till the graceful termination period ends, then they will get terminated by the SIGKILL signal. (reference: kubernetes docs)

Kranthiveer Dontineni
  • 1,056
  • 1
  • 10
  • Can't we destory these containers without waiting for terminationGracePeriodSeconds gets over ?. In my case, i have to set more time for graceful termination period. – sethu ram May 24 '23 at 03:45
  • You can use the `flag --force` to terminate a Pod by force. However it will terminate all the containers forcefully. So, it is not a suggested way of deleting the pods if your containers doesn't contain much data and if you can make sure they are not performing any write operations at that time, you can go with forceful deletion/termination as this will not cause any data corruption. – Kranthiveer Dontineni May 24 '23 at 04:37
  • No other way without force ? As mentioned, it is in pending state so preStop lifecycle also not possible to hit ? – sethu ram May 25 '23 at 16:30
  • @sethuram Can you please brief the scenario i.e is it for maintenance or for changing the deployment or for scenarios such as OOM, resource utilization or any other pod eviction or node eviction cases. – Kranthiveer Dontineni May 26 '23 at 08:03
  • At the time of probe failure, container will be in pending state and not handling any signals. In my case, for normal positive cases I have to set terminationGracePeriodSeconds to 500. Since this period is common for failure and success cases. "How do i avoid this period (500) in probe failure case ?" – sethu ram May 31 '23 at 05:00
  • In success cases, Container will be in UP and READY which will handle signals and it may go down within 50 or 100s, No need to wait untill the 500s to gets over. But failure case am not able to handle. => As we are doing some actions in preStop may require this 500s. – sethu ram May 31 '23 at 05:03