AirFlow not running schedule

Question

I have a very simple dag which is supposed to run every Monday at 19:10. The dag is as follows:

from airflow.models import DAG
from airflow.utils.dates import days_ago
from airflow.operators.bash_operator import BashOperator


args = {'owner': 'AirFlow'}

with DAG(dag_id='schedule_test_weekly', default_args=args, schedule_interval="20 21 * * 1", tags=['Scheduler test'],
         start_date=days_ago(2), catchup=False) as dag:
    test_scheduler_health = BashOperator(task_id="test_test_weekly", bash_command="echo Hello World", dag=dag)

But, it does not trigger. If I change the schedule_interval to schedule_interval="*/5 * * * *" it will run every 5 minutes without a problem. But I can't have it run every Monday at the said time. I checked many guides and posts online. Most mention the executer should be LocalExecutor and database connection should be set to postgres. I have these configs:

sql_alchemy_conn = postgresql+psycopg2://airflow:pass2@localhost:5432/airflow
sql_alchemy_pool_enabled = True
sql_alchemy_pool_size = 10
executor = LocalExecutor

I also used cronhub to check my cron schedule. I think it is correct. Am I missing anything or doing something wrong?

Try checking this [answer](https://stackoverflow.com/a/66566864/10569220). There is no problem with the executor or the database, is just how scheduling works on Airflow. By the way, I think cron for every monday at 19:10 is: " 10 19 * * 1" — NicoE, Apr 06 '21 at 13:01

score 1 · Accepted Answer · answered Apr 06 '21 at 14:07

1

The problem in your code is not using start_date correctly.

If you want a weekly job to run today (Monday), the start_date needs to be last week (Monday). The reason explained in this question.

So (assuming today is 05-Apr-2021) your DAG needs to be:

with DAG(dag_id='schedule_test_weekly',
         default_args=args,
         schedule_interval="10 19 * * 1",
         tags=['Scheduler test'],
         start_date=datetime(2021, 3, 29), # Monday 
         catchup=False
         ) as dag:

The first run will start on 2021-04-05 (today) with execution_date of 2021-03-29.

The next run will start on 2021-04-12 with execution_date of 2021-04-05 and following on with that pattern as weekly job.

answered Apr 06 '21 at 14:07

Elad Kalif

14,110
2
17
49

I tried. Still not working. I went as far as a month ago. I discovered that if I pass one day in the schedule_interval it wont work. Could there be a bug in AirFlow? – Mansour.M Apr 06 '21 at 20:43
1

Try changing the dag_id – Elad Kalif Apr 07 '21 at 04:34
Wow! That seems to have fixed it! Thanks! But do you know what the problem with the previous dag_id is? – Mansour.M Apr 07 '21 at 14:09
Also, can you add that to the response so I can choose it as the correct response? – Mansour.M Apr 07 '21 at 14:19
1

It shouldnt happen but sometimes in older versions you must change the dag_id when you change the interval, its a side effect of the scheduling mechanism that is based on interval since the last run (your last run wasnt on Monday so it confuses the scheduler) so new dag_id gives a fresh start. I dont add it to the answer because this shouldnt happen in updated Airflow versions. – Elad Kalif Apr 07 '21 at 16:06

AirFlow not running schedule

1 Answers1