4

I've been playing around with SubDAGs. A big problem I've faced is whenever something within the SubDAG fails, and I re-run things by hitting Clear, only the cleared task will re-run; the success does not propagate to downstream tasks in the SubDAG and get them running.

How do I re-run a failed task in a SubDAG such that the downstream tasks will flow correctly? Right now, I have to literally re-run every task in the SubDAG that is downstream of the failed task.

I think I followed the best practices of SubDAGs; the SubDAG inherits the Parent DAG properties wherever possible (including schedule_interval), and I don't turn the SubDAG on in the UI; the parent DAG is on and triggers it instead.

sophros
  • 14,672
  • 11
  • 46
  • 75
selwyth
  • 2,417
  • 16
  • 19
  • Possible duplicate of [How to run one airflow task and all its dependencies?](https://stackoverflow.com/questions/42633892/how-to-run-one-airflow-task-and-all-its-dependencies) – sophros Oct 29 '19 at 12:34

1 Answers1

0

A bit of a workaround but in case you have given your tasks task_id-s consistently you can try the backfilling from Airflow CLI (Command Line Interface):

airflow backfill -t TASK_REGEX ... dag_id

where TASK_REGEX corresponds to the naming pattern of the task you want to rerun and its dependencies.

(remember to add the rest of the command line options, like --start_date).

sophros
  • 14,672
  • 11
  • 46
  • 75