0

I see in the Airflow 2 SubdagOperator documentation link that using mode = reschedule we can get rid of potential deadlock. To my understanding it is not a param which can be be passed with list of other params. If anyone has used this let me know how to incorporate it in SubdagOperator.

codninja0908
  • 497
  • 8
  • 29

1 Answers1

1

Technically, a SubDagOperator is a sensor, which can take an argument mode="reschedule". The default mode poke keeps a slot open, which can potentially lead to a deadlock situation in case you're using lots of sensors. Mode reschedule instead stops the process and creates a new process on every check, not causing a situation where all slots are occupied by sensors waiting on each other.

SubDagOperator(task_id="foobar", ..., mode="reschedule")

With that said, the SubDagOperator is deprecated since Airflow 2.0 and it is advised to use TaskGroups. TaskGroups are a visual way to group together tasks within a DAG (tutorial here: https://www.astronomer.io/guides/task-groups).

Alternatively, you can use the TriggerDagRunOperator to trigger another DAG (tutorial: https://www.astronomer.io/guides/cross-dag-dependencies).

Bas Harenslak
  • 2,591
  • 14
  • 14
  • Thanks for the information. Regarding TaskGroup, it has a downside to its UI. Unlike other operators it does not provide "Clear", "Run" operations to run all the tasks underneath in one go rather one has to clear, run each individual task. In our case we have multiple(> 15) task grouped together in one Subdag. Therefore, we cannot move towards TaskGroup for now – codninja0908 Dec 29 '21 at 12:59
  • Fair point, there is an issue: https://github.com/apache/airflow/issues/14529 although it doesn't seem like there's any activity on it at the moment. – Bas Harenslak Dec 29 '21 at 13:49