Kubernetes Cron Job Terminate Pod before creation of next schedule

Question

I have a Kubernetes Cron Job for running a scheduled task every 5 minutes. I want to make sure that when a new pod is created at next schedule time, the earlier pod should have been terminated. The earlier pod should get terminated before creation of new. Can Kubernetes terminate the earlier pod before creation of new?

My yaml is:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-scheduled
spec:
  schedule: "*/5 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cmm-callout
            env:
              - name: SCHEDULED
                value: "true"
            livenessProbe:
              httpGet:
                path: /myapp/status
                port: 7070
                scheme: HTTPS
              initialDelaySeconds: 120
              timeoutSeconds: 30
              periodSeconds: 120                
            image: gcr.io/projectid/folder/my-app:9.0.8000.34
          restartPolicy: Never

How can I make sure the earlier pod is terminated before new is created?

score 9 · Answer 1 · answered Aug 05 '19 at 10:47

If i understood your case correctly (the earlier pod should have been terminated before creation of new one).

1. Please use spec.jobTemplate.spec.activeDeadlineSeconds instead.

By setting this parameter once a Job reaches activeDeadlineSeconds - all of running Pods will be terminated and the Job status will become type: Failed with reason DeadlineExceeded.

example:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      activeDeadlineSeconds: 60
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster && sleep 420
          restartPolicy: Never

2. The second solution is to set-up concurrencyPolicy. and replace the currently running job with a new job.

example:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/2 * * * *"
  concurrencyPolicy: Replace
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster && sleep 420
          restartPolicy: Never

Resources:

score 2 · Answer 2 · answered Jul 30 '19 at 06:47

Did you try to set the concurrencyPolicy to Replace? Forbid means to skip the new job run if the previous one hasn't finished yet.

https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#concurrency-policy

Allow (default): The cron job allows concurrently running jobs

Forbid: The cron job does not allow concurrent runs; if it is time for a new job run and the previous job run hasn’t finished yet, the cron job skips the new job run

Replace: If it is time for a new job run and the previous job run hasn’t finished yet, the cron job replaces the currently running job run with a new job run

score 2 · Answer 3 · answered Sep 03 '20 at 09:52

I'm using Mark's solution with spec.jobTemplate.spec.activeDeadlineSeconds.

Just that there's one more thing into it. From the K8S docs:

Once a Job reaches activeDeadlineSeconds, all of its running Pods are terminated and the Job status will become type: Failed with reason: DeadlineExceeded.

What actually happens when the pod is terminated is that K8S triggers a SIGTERM against the POD's container process pid 0. It's not waiting for the actual process to terminate. If your container does not gracefully terminate, it's going to stay into terminating state for 30 seconds, after which K8S triggers a SIGKILL. In the meantime, K8S potentially schedules another pod so the terminating one overlaps with the new scheduled one for at most 30 seconds.

This is easily reproducible with this CronJob definition:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cj-sleep
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 5
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      activeDeadlineSeconds: 50
      template:
        metadata:
          creationTimestamp: null
        spec:
          containers:
          - command:
                - "/usr/local/bin/bash"
                - "-c"
                - "--"
            args:
                - "tail -f /dev/null & wait $!"
            image: bash
            imagePullPolicy: IfNotPresent
            name: cj-sleep
          dnsPolicy: ClusterFirst
          restartPolicy: OnFailure
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
  schedule: '* * * * *'
  startingDeadlineSeconds: 100
  successfulJobsHistoryLimit: 5

this is how the scheduling happens:

while true; do date; kubectl get pods -A | grep cj-sleep; sleep 1; done
    
Thu Sep  3 09:50:51 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Running            0          49s
Thu Sep  3 09:50:53 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating        0          50s
Thu Sep  3 09:50:54 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating        0          51s
Thu Sep  3 09:50:55 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating        0          52s
Thu Sep  3 09:50:56 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating        0          54s
Thu Sep  3 09:50:58 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating        0          56s
Thu Sep  3 09:51:00 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating        0          57s
Thu Sep  3 09:51:01 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating        0          58s
Thu Sep  3 09:51:02 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating        0          59s
Thu Sep  3 09:51:03 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating         0          60s
default                                     cj-sleep-1599126660-l69gd                                         0/1     ContainerCreating   0          0s
Thu Sep  3 09:51:04 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating         0          61s
default                                     cj-sleep-1599126660-l69gd                                         0/1     ContainerCreating   0          1s
Thu Sep  3 09:51:05 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         1/1     Terminating        0          62s
default                                     cj-sleep-1599126660-l69gd                                         1/1     Running            0          2s
    
    ....
Thu Sep  3 09:51:29 UTC 2020
default                                     cj-sleep-1599126600-kzzxg                                         0/1     Terminating        0          86s
default                                     cj-sleep-1599126660-l69gd                                         1/1     Running            0          26s
Thu Sep  3 09:51:30 UTC 2020
default                                     cj-sleep-1599126660-l69gd                                         1/1     Running            0          28s
Thu Sep  3 09:51:32 UTC 2020
default                                     cj-sleep-1599126660-l69gd                                         1/1     Running            0          29s

There is a detail specific to init 0 processes, they don't handle SIGTERM by default, you have to provide your own handler. In case of bash, it's by adding a trap:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cj-sleep
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 5
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      activeDeadlineSeconds: 50
      template:
        metadata:
          creationTimestamp: null
        spec:
          containers:
          - command:
                - "/usr/local/bin/bash"
                - "-c"
                - "--"
            args:
                - "trap 'exit' SIGTERM; tail -f /dev/null & wait $!"
            image: bash
            imagePullPolicy: IfNotPresent
            name: cj-sleep
          dnsPolicy: ClusterFirst
          restartPolicy: OnFailure
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
  schedule: '* * * * *'
  startingDeadlineSeconds: 100
  successfulJobsHistoryLimit: 5

And now this is how the scheduling happens:

Thu Sep  3 09:47:54 UTC 2020
default                                     cj-sleep-1599126420-sm887                                         1/1     Terminating        0          52s
Thu Sep  3 09:47:56 UTC 2020
default                                     cj-sleep-1599126420-sm887                                         0/1     Terminating        0          54s
Thu Sep  3 09:47:57 UTC 2020
default                                     cj-sleep-1599126420-sm887                                         0/1     Terminating        0          55s
Thu Sep  3 09:47:58 UTC 2020
default                                     cj-sleep-1599126420-sm887                                         0/1     Terminating        0          56s
Thu Sep  3 09:47:59 UTC 2020
default                                     cj-sleep-1599126420-sm887                                         0/1     Terminating        0          57s
Thu Sep  3 09:48:00 UTC 2020
default                                     cj-sleep-1599126420-sm887                                         0/1     Terminating        0          58s
Thu Sep  3 09:48:01 UTC 2020
Thu Sep  3 09:48:02 UTC 2020
default                                     cj-sleep-1599126480-rlhlw                                         0/1     ContainerCreating   0          1s
Thu Sep  3 09:48:04 UTC 2020
default                                     cj-sleep-1599126480-rlhlw                                         0/1     ContainerCreating   0          2s
Thu Sep  3 09:48:05 UTC 2020
default                                     cj-sleep-1599126480-rlhlw                                         0/1     ContainerCreating   0          3s
Thu Sep  3 09:48:06 UTC 2020
default                                     cj-sleep-1599126480-rlhlw                                         1/1     Running            0          4s

Kubernetes Cron Job Terminate Pod before creation of next schedule

3 Answers3

Linked