0


I cannot understand how scheduler works, I created some dags with:

schedule_interval="0 21 * * *",
start_date=pendulum.datetime(2022, 5, 3, tz="UTC"),
catchup=False,
"max_active_runs": 1,


I activated the dag yesterday afternoon around 15.00 and I executed manually the dags to verify they works.
This morning I checked the dag execution and I saw the the dags have not been executed yesterday (2022-05-03 at 21.00) as expected (this is the next run date reported by airflow) why?
What am I wronging?
I am using airflow 2.2.4 Thanks

afmulone
  • 177
  • 1
  • 1
  • 8

1 Answers1

1

Manual runs start as soon as you trigger it. Scheduled runs are subject to Airflow scheduling mechanisem. I explained it in Problem with start date and scheduled date in Apache Airflow

In your case start date of 2022-05-03 with interval of 0 21 * * * means that the first run will start on 2022-05-04 21:00. The execution_date of this run will be 2022-05-03 21:00. So if you wanted the first run to start on 2022-05-03 21:00 you need to set:

start_date=pendulum.datetime(2022, 5, 2, tz="UTC"),
schedule_interval="0 21 * * *",

Alternatively, if you want to specify exact run times you will need to use Timetables then you can customize DAG Scheduling with Timetables.

Elad Kalif
  • 14,110
  • 2
  • 17
  • 49
  • Ok, so today is the 2022-05-04, I want that the today ad 21.00 the dags shall be executed with the current date I should set: start_date=pendulum.datetime(2022, 5, 4, tz="UTC")? Is it correct? – afmulone May 04 '22 at 09:42
  • 1
    You are confusing between when the DAG is executed with what data interval you process. I don't know what your DAG is doing but you can get any date reference you need for the actual DAG work if you will use macros https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html – Elad Kalif May 04 '22 at 09:48
  • Yes, I have a lot of confusion and reading the documentation I cannot understand until now, I want to schedule a dag as crontab with logical date that is equal to the time when the dag is triggered by the scheduler, this is my objective. – afmulone May 04 '22 at 09:56
  • But Airflow is not crontab... The scheduling logic is as I explained. You can not change it. – Elad Kalif May 04 '22 at 10:36