1

I have cronjob that is defined by this manifest:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: trigger
spec:
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 5
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      activeDeadlineSeconds: 50
      backoffLimit: 1
      parallelism: 1
      template:
        spec:
          containers:
          - env:
            - name: ApiKey
              valueFrom:
                secretKeyRef:
                  key: apiKey
                  name: something
            name: trigger
            image: curlimages/curl:7.71.1
            args:         
            - -H 
            - "Content-Type: application/json"
            - -H
            - "Authorization: $(ApiKey)"
            - -d 
            - '{}'
            - http://url
          restartPolicy: Never

It sort of works, but not 100%. For some reason it runs 10 jobs, then it pauses for 5-10 minutes or so and then run 10 new jobs. No errors are reported, but we don't understand why it pauses.

Any ideas on what might cause a cronjob in kubernetes to pause?

Tomas Jansson
  • 22,767
  • 13
  • 83
  • 137
  • Have you tried describing the `CronJob` and looking at related `Job`s and `Event`s? – BogdanL Aug 11 '20 at 23:59
  • @BogdanL, of course. Couldn't find anything strange, so really weird. – Tomas Jansson Aug 12 '20 at 11:34
  • Hi, i've also tested it on my cluster and i got the same result as you. I've seen in the pod-logs of `kube-controller-manager` some errors regarding the Job, can you check if you have also some errors in there? `Set or decrease .spec.startingDeadlineSeconds or check clock skew` – CLNRMN Aug 14 '20 at 08:14

1 Answers1

0

The most common problem of running CronJobs on k8s is spawning to many pods which consume all cluster resources. It is very important to set proper CronJob limitations. So try to set memory limits for pods.

Also speaking about concurrencyPolicy you set concurrencyPolicy param to Forbid which means that the cron job does not allow concurrent runs; if it is time for a new job run and the previous job run hasn't finished yet, the cron job skips the new job run.

The .spec.concurrencyPolicy field is optional. It specifies how to treat concurrent executions of a job that is created by this cron job. There are following concurrency policies:

  • Allow (default): The cron job allows concurrently running jobs
  • Forbid: explained above
  • Replace: If it is time for a new job run and the previous job run hasn't finished yet, the cron job replaces the currently running job run with a new job run

Try to change policy to allow or replace according to your needs.

Speaking about a non-parallel Job, you can leave .spec.parallelism unset. When it is unset, it is defaulted to 1.

Take a look: cron-jobs-running-for-one-cron-execution-point-in-kubernetes, cron-job-limitations, cron-jobs.

Malgorzata
  • 6,409
  • 1
  • 10
  • 27
  • The spawning of pods couldn't be an issue, I added even more cronjobs trying to reproduce it in a reliable fashion.. instead it all started working as expected. Not sure why the `concurrencyPolicy` set to something else would have an effect when the actual cronjob finishes in a matter of seconds. – Tomas Jansson Aug 12 '20 at 11:33
  • So now everything is sort of working, but not sure why. I did increase the `successfulJobsHistoryLimit`, `startingDeadlineSeconds` and `activeDeadlineSeconds`, but I can't see how any of those properties would cause the job to pause on a regular schedule. – Tomas Jansson Aug 12 '20 at 11:35
  • Speaking about concurrencyPolicy you have said that your cronjob runs 10 jobs, then it pauses for 5-10 minutes and then run 10 new jobs, this flag may have affect on it, also did you check cluster resources consumption ? – Malgorzata Aug 19 '20 at 15:04