4

I have configured a CronJon in K8S that creates a pod every 10 minutes. The running container can complete in anything from 5 minutes to 2 hours, no way to tell.

Sometimes, when the computation is taking very long, i end up with far too many pods created.

Is there a way to limit the number of running pods, instances of the specific CronJob?

EDIT

To clarify, I want concurrent runs, i want to limit the number of running concurrent instances to a certain large number N.

Ylli Prifti
  • 323
  • 3
  • 11

2 Answers2

1

Refer 

 kubectl explain cronjob.spec.jobTemplate.spec

This will give you "parallelism" key, you can map the "parallelism" to the number 'N', as required by you.

This will allow 'N' number of pods to run in parallel.

I guess this helps.

Tushar Mahajan
  • 2,044
  • 1
  • 7
  • 18
  • Maybe my question was not clear - I added one more sentence to clarify. I want concurrent runs, i want to limit the number of running concurrent instances to a certain large number say 50. – Ylli Prifti Jan 02 '20 at 10:13
  • 1
    thanks again - I thought parallelism indicates how many pods to create at each run. I double checked it and in fact, parallelism does not limit but rather creates N pods at each run. Having put N at 50 i ended up with 200 instances very fast. However i managed to find the solution / posting it here. – Ylli Prifti Jan 02 '20 at 14:04
1

Following up from @Tushar's reply, in combination with the information from this post: https://www.alibabacloud.com/blog/kubernetes-cronjobs---part-2-parallelism_595022

The solution is a combination of three settings, that are counter indicative if looked separately.

To achieve a maximum number of 10 running pods, instances of a cronjob i did the following 

      parallelism: 10        
      completions: 8

  concurrencyPolicy: Forbid

Each time the tick will hit, CronJob will create 10 parallel pods, but on the next tick - unless 8 of the previous pods have completed, will not trigger the run due to concurrencyPolicy: Forbid

This also means that the tick will be skipped if say only three instances are still running, but this is close enough to what i was looking for.

Ylli Prifti
  • 323
  • 3
  • 11
  • But if `completions` is set lower than `parallelism`, then at each cycle you can unwillingly increase the number of running pods by up to the difference between these two parameters? – mirekphd Jun 21 '21 at 18:56
  • Yes, in this case, you could end up with 12 running instances (8 completed, next batch will run, two still running + 10 from the next batch). Having 12 running instances is OK in my case, also they will be at different execution stages and this is absolutely fine for my case. – Ylli Prifti Apr 15 '22 at 17:58