4

Basically I am trying to do is play around with pod lifecycle and check if we can do some cleanup/backup such as copy logs before the pod terminates.

What I need : Copy logs/heapdumps from container to a hostPath/S3 before terminating

What I tried:

I used a preStop hook with a bash command to echo a message (just to see if it works !!). Used terminationGracePeriodSeconds with a delay to preStop and toggle them to see if the process works. Ex. keep terminationGracePeriodSeconds:30 sec (default) and set preStop command to sleep by 50 sec and the message should not be generated since the container will be terminated by then. This works as expected.

My questions:

  • what kind of processes are allowed(recommended) for a preStop hook? As copying logs/heapdumps of 15 gigs or more will take a lot of time. This time will then be used to define terminationGracePeriodSeconds
  • what happens when preStop takes more time than the set gracePeriod ? (in case logs are huge say 10 gigs)
  • what happens if I do not have any hooks but still set terminationGracePeriodSeconds ? will the container remain up until that grace time ?

I found this article which closely relates to this but could not follow through https://github.com/kubernetes/kubernetes/issues/24695

All inputs appreciated !!

Prathamesh dhanawade
  • 493
  • 5
  • 27

1 Answers1

4

what kind of processes are allowed(recommended) for a preStop hook? As copying logs/heapdumps of 15 gigs or more will take a lot of time. This time will then be used to define terminationGracePeriodSeconds

Anything goes here, it's more of an opinion and how you would like your pods to linger around. Another option is to let your pods terminate and store your data in some place (i.e, AWS S3, EBS) where data will persist past the pod lifecycle then use something like Job to clean up the data, etc.

what happens when preStop takes more time than the set gracePeriod? (in case logs are huge say 10 gigs)

Your preStop will not complete which may mean incomplete data or data corruption.

what happens if I do not have any hooks but still set terminationGracePeriodSeconds ? will the container remain up until that grace time ?

This explains would be the sequence:

  • A SIGTERM signal is sent to the main process in each container, and a “grace period” countdown starts.
  • If a container doesn’t terminate within the grace period, a SIGKILL signal will be sent and the container.
Rico
  • 58,485
  • 12
  • 111
  • 141
  • Thanks. Also, say I have gracePeriod: 300 secs (i.e. 5 min) and a preStop hook that completes within 10 seconds (guaranteed). My thought was that the preStop hook takes control and sends a SIGKILL once it finishes so that the container doesn't linger around for the rest of the 290 secs(which potentially saves time and resources) ?!! – Prathamesh dhanawade Feb 13 '19 at 23:44
  • hmm, I don't think so – Rico Feb 14 '19 at 06:46
  • I found out that kubelet is responsible for transferring control to and fro preStop hook (https://blog.gruntwork.io/gracefully-shutting-down-pods-in-a-kubernetes-cluster-328aecec90d) and the kubelet finally sends the SIGTERM to delete the pod. Also, one other case: what happens when there is no grace period set but a preStop hook is configured to transfer logs ?? – Prathamesh dhanawade Feb 21 '19 at 16:56
  • 1
    The default grace period is 30 seconds: https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods – Rico Feb 21 '19 at 17:53
  • So If preStop ends before gracePeriod (say 10 seconds while gracePeriod default is 30 secs) then the pod stays up till the rest of 20 secs and then terminates. Is there a way to terminate the pod as soon as preStop completes so that it is easier to define gracePeriod (we can set gracePeriod large enough to let preStop complete while not have to wait for gracePeriod to end in case preStop completes way too earlier !!)..?? – Prathamesh dhanawade Feb 22 '19 at 20:22
  • 1
    If both application and preStop can properly handle signals, it won't wait gracePeriod as you expected. – ccshih Apr 22 '19 at 09:24
  • This is incorrect. SIGTERM is only sent after the preStop hook. – Code Feb 27 '20 at 20:18
  • 1
    @Code thanks for pointing out. I've removed the preStop hook bullet point. – Rico Feb 27 '20 at 21:32
  • Also, I've removed the original link pointing to the explanation as it's now broken. – Rico Feb 27 '20 at 22:40