Set Workflow Job Concurrency Limit in Databricks

Question

I need a job to be triggered every 5 minutes. However, if that job is already running, it must not be triggered again until that run is finished. Hence, I need to set the maximum run concurrency for that job to only one instance at a time.

What needs to be configured to achieve that? Any recommendations and pointers would be much appreciated

Best Regards

Answer 1

You want max_concurrent_runs = 1 on the job.

Refer to Jobs API.

Example payload:

{
  "name": "Some job",
  "tasks": [...],
  "job_clusters": [...],
  "schedule": {
    "quartz_cron_expression": "0 0/5 * * * ?",  # Every 5 minutes
    "timezone_id": "Europe/London"
  }
  "max_concurrent_runs": 1,  # Only 1 concurrent run allowed
  "format": "MULTI_TASK",
}

It's also possible to set this in UI.