What happens if run same dag multiple times while already running?

Question

What happens if the same dag is triggered concurrently (or such that the run times overlap)?

Asking because recently manually triggered a dag that ended up still being running when its actual scheduled run time passed, at which point, from the perspective of the web-server UI, it began running again from the beginning (and I could no longer track the previous instance). Is this just a case of that "run instance" overloading the dag_id or is the job literally restarting (ie. the previous processes are killed)?

Answer 1

As I understand it depends on how it was triggered and if the DAG has a schedule. If it's based on the schedule defined in the DAG say a task to run daily it is incomplete / still working and you click the rerun then this instance of the task will be rerun. i.e the one for today. Likewise if the frequency were any other unit of time.

• If you wanted to rerun other instances you need to delete them from the previous jobs as described by @lars-haughseth in a different question. airflow-re-run-dag-from-beginning-with-new-schedule
• If you trigger a DAG run then it will get the triggers execution timestamp and the run will be displayed separately to the scheduled runs. As described in the documentation here. external-triggers documentation

Note that DAG Runs can also be created manually through the CLI while running an airflow trigger_dag command, where you can define a specific run_id. The DAG Runs created externally to the scheduler get associated to the trigger’s timestamp, and will be displayed in the UI alongside scheduled DAG runs.

In your instance it sounds like the latter. Hope that helps.