5

We have one airflow DAG which is accepting input from user and performing some task. we want to run same DAG simultaneous with different input from user. we found multiple links for simultaneous task run but not able to get info about simultaneous run. so if we triggered DAG with two diff inputs from cli then its running fine with two instances but just want to understand that both instances are running independently or its waiting for one instance to finish and then trigger the other ?

Yug
  • 105
  • 1
  • 9

1 Answers1

5

All of what you mention can be done. Tasks can be executed in parallel.

It's just the proper configuration of max_active_runs wait_for_downstream and depends_on_past

DAG parameters:

max_active_runs - maximum number of active DAG runs

if you just want the DAGs to be able to execute two jobs in parallel (with no conditions between two distinct runs) then set max_active_runs=2

Operators parameters:

wait_for_downstream - when set to true, an instance of task X will wait for tasks immediately downstream of the previous instance of task X to finish successfully or be skipped before it runs

depends_on_past - when set to true, task instances will run sequentially and only if the previous instance has succeeded or has been skipped.

Elad Kalif
  • 14,110
  • 2
  • 17
  • 49
  • Thanks @Elad. What is the default value for max_active_runs ? if i didn't pass anything – Yug Jan 06 '22 at 10:09
  • Default value for max_active_runs is 16 https://github.com/apache/airflow/blob/8cf3d120ff44c6604e330cc9c4e7945b3f1edc6c/airflow/config_templates/default_airflow.cfg#L132 – Elad Kalif Jan 06 '22 at 13:33