8

Backstory: I was running an Airflow job on a daily schedule, with a start_date of July 1, 2019. The job gathered requested each day's data from a third party, then loaded that data into our database.

After running the job successfully for several days, I realized that the third party data source only refreshed their data once a month. As such, I was simply downloading the same data every day.

At that point, I changed the start_date to a year ago (to get previous months' info), and changed the DAG's schedule to run once a month.

How do I (in the airflow UI) restart the DAG completely, such that it recognizes my new start_date and schedule, and runs a complete backfill as if the DAG is brand new?

(I know this backfill can be requested via the command line. However, I don't have permissions for the command line interface and the admin is unreachable.)

Ashley O
  • 1,130
  • 3
  • 21
  • 34

1 Answers1

7

Click on the green circle in the Dag Runs column for the job in question in the web interface. This will bring you to a list of all successful runs.

Tick the check mark on the top left in the header of the list to select all instances, then in the menu above it choose "With selected" and then "Delete" in the drop down menu. This should clear all existing dag run instances.

If catchup_by_default is not enabled on your Airflow instance, make sure catchup=True is set on the DAG until it has finished catching up.

Lars Haugseth
  • 14,721
  • 2
  • 45
  • 49
  • 1
    This worked and was correct. I needed to not only clear, but delete the task runs. Thank you for this insight – Ashley O Jul 09 '19 at 17:32
  • 1
    It sounds like this will rerun all DagRuns including the ones that were already run. Is there a way to only run the new tasks between the new start date and the old start date? – chris.mclennon Jul 10 '19 at 16:53
  • Is there a way to run it again without deleting previous DAG runs? I am experimenting with some data storage strategies and I want to see if it time for the DAG runs varies with it. After all I want to improve the processing time. – harvpan Feb 08 '21 at 19:34