8

EDIT: Question is solved, it was my mistake, i simply used the wrong cron settings. I assumed "* 2 * * *" would only run once per day at 2, but in fact it runs every minute past the hour 2. So Kubernetes behaves correctly.

I keep having multiple jobs running at one cron execution point. But it seems only if those jobs have a very short runtime. Any idea why this happens and how I can prevent it? I use concurrencyPolicy: Forbid, backoffLimit: 0 and restartPolicy: Never.

Example for a cron job that is supposed to run once per day, but runs multiple times just after its scheduled run time: 

job-1554346620                   1/1           11s        4h42m   
job-1554346680                   1/1           11s        4h41m                     
job-1554346740                   1/1           10s        4h40m

Relevant config:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: job
spec:
  schedule: "* 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: job
              image: job_image:latest
              command: ["rake", "run_job"]
          restartPolicy: Never
          imagePullSecrets:
            - name: regcred
      backoffLimit: 0
Nico
  • 881
  • 1
  • 6
  • 19

2 Answers2

18

The most common problem of running CronJobs on k8s is:

spawning to many pods which consume all cluster resources

It is very important to set proper CronJob limitations

If you are not sure what you need - just take this example as a template:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-first-conjob
  namespace: devenv-admitriev
spec:
  schedule: "*/10 * * * *" # MM HH DD MM WKD -- Minutes, Hour, Day, Month, Weekday (eg. Sun, Mon)
  successfulJobsHistoryLimit: 3 # how many completed jobs should be kept
  failedJobsHistoryLimit: 1 # how many failed jobs should be kept
  suspend: false # Here you can suspend cronjob without deliting it
  concurrencyPolicy: Forbid # Choose Forbid if you don't want concurrent executions of your Job

  # The amount of time that Kubernetes can miss and still start a job.
  # If Kubernetes missed too many job starts (100)
  # then Kubernetes logs an error and doesn’t start any future jobs.
  startingDeadlineSeconds: 300 # if a job hasn't started in this many seconds, skip
  jobTemplate:
    spec:
      parallelism: 1 # How many pods will be instantiated at once.
      completions: 1 # How many containers of the job are instantiated one after the other (sequentially) inside the pod.
      backoffLimit: 3 # Maximum pod restarts in case of failure
      activeDeadlineSeconds: 1800 # Limit the time for which a Job can continue to run
      template:
        spec:
          restartPolicy: Never # If you want to restart - use OnFailure
          terminationGracePeriodSeconds: 30
          containers:
          - name: my-first-conjob
            image: busybox
            command:
              - /bin/sh
            args:
              - -c
              - date; echo sleeping....; sleep 90s; echo exiting...;
            resources:
              requests:
                memory: '128Mi'
              limits:
                memory: '1Gi'
Alexey Dmitriev
  • 361
  • 2
  • 6
  • 1
    Excellent! But, as the docs warn, it's a risk limitation, not a risk-free guarantee: "even if you specify `.spec.parallelism` = 1 and `.spec.completions` = 1 and `.spec.template.spec.restartPolicy` = "Never", the same program may sometimes be started twice." – mirekphd Jun 21 '21 at 19:15
2

Hi it's not clear what you expected - looking into the question but if I understand correctly you mean not running all cronjobs at the same time:
1. First option - it's to change their schedule time,
2. Second option try to use in your spec template other options like - Parallel Jobs - described: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/

"For a work queue Job, you must leave .spec.completions unset, and set .spec.parallelism to a non-negative integer"


  jobTemplate:
    spec:
      parallelism: 1
      template:

To recreate this task please provide more details.

In addition for "Jobs History" by default successfulJobsHistoryLimit and failedJobsHistoryLimit are set to 3 and 1 respectively.
Please take at: https://kubernetes.io/docs/tasks/job/ If you are interested you can set-up limit in "spec" section: 


successfulJobsHistoryLimit: 1

failedJobsHistoryLimit: 1

Hope this help.

Mark
  • 3,644
  • 6
  • 23
  • Hi Hanx, there should only be running one job for each cron execution, not 3 or more. If you look at that schedule, the job should run only once per day. But Kubernetes runs it multiple times for some reason, as you can see in the log snippet i provided. It runs multiple times with only a one minute interval: it ran 4h40m, 4h41m, 4h42m ago. – Nico Apr 05 '19 at 14:34
  • Edited the question slightly to make it more clear. – Nico Apr 05 '19 at 15:36
  • 1
    Sorry, my mistake! I got the cron syntax wrong. schedule: "* 2 * * *" means once every minute when the hour is 2. So Kubernetes behaves correctly. I simply should have set it to e.g. "0 2 * * *". Thanks Hanx for giving some more infos about Kubernetes Jobs! – Nico Apr 11 '19 at 05:31