0

Could anyone please let me know whether I can spin up many (say, around 10 concurrent DAGs) concurrent DAGs simultaneously in parallel at the same time? The parameters for these DAGs will be the same but with different values. Any ideas or suggestions would be greatly helpful as stuck on this. Thanks!

testbg testbg
  • 193
  • 2
  • 11
  • You should be able to do that. Examples: [Trigger Controller DAG](https://github.com/apache/airflow/blob/main/airflow/example_dags/example_trigger_controller_dag.py), [Trigger Target DAG](https://github.com/apache/airflow/blob/main/airflow/example_dags/example_trigger_target_dag.py). Using the concept in this, you should be able to trigger as many loads as you want. Just be careful about the amount of resources you spend per load – Tameem Jan 14 '22 at 04:42
  • If you are able to trigger and the tasks are not running concurrently, checkout the concurrency settings in the airflow.cfg file. – Tameem Jan 15 '22 at 06:32
  • @Tameem, Thanks! Is there any sample or code snippet that you can share? I'm currently using the below. Will this work? DAG1 >> (DAG2, DAG3, DAG4,......DAG30) – testbg testbg Jan 18 '22 at 21:12

0 Answers0