Stack Overflow
Stack Overflow

Questions tagged [airflow-scheduler]

The Apache Airflow scheduler monitors all tasks and all DAGs, and triggers the task instances whose dependencies have been met, and Apache Airflow is a platform to programmatically author, schedule and monitor workflows.

1257 questions
7
votes
6 answers

How to instruct airflow to backfill from most recent to oldest

I have an Airflow DAG scheduled to run daily. When I start a backfill for the last month, Airflow will start processing the runs from oldest to newest. As a single run takes a couple of hours, which means that when a new run becomes available (a day…
zeebonk
  • 4,864
  • 4
  • 21
  • 31
7
votes
3 answers

Efficient ways of implementing waiting till a certain criterion is met in Airflow

Sensors in Airflow - are a certain type of operator that will keep running until a certain criterion is met but they consume a full worker slot. Curious if people have been able to reliably use more efficient ways of implementing this. A few ideas…
sharky
  • 327
  • 4
  • 13
7
votes
2 answers

Airflow: re execute the jobs of a DAG for the past n days on a daily basis

I have scheduled the execution of a DAG to run daily. It works perfectly for one day. However each day I would like to re-execute not only for the current day {{ ds }} but also for the previous n days (let's say n = 7). For example, in the next…
lucacerone
  • 9,859
  • 13
  • 52
  • 80
6
votes
2 answers

Airflow - prevent dagrun from immediately running after deployment/unpause

Seems there there has been previous discussion about this. How do i stop airflow running a task the first time when i unpause it? https://groups.google.com/g/cloud-composer-discuss/c/JGtmAd7xcsM?pli=1 When I deploy a dag to run at a specific time…
user3240688
  • 1,188
  • 3
  • 13
  • 34
6
votes
1 answer

How do I stop Apache Airflow running a task the first time when I unpause it?

I have a DAG. Here is a sample of the parameters. dag = DAG( 'My Dag', default_args=default_args, description='Cron Job : My Dag', schedule_interval='45 07 * * *', # start_date=days_ago(0), start_date = datetime(2021, 4, 6,…
raaj
  • 403
  • 1
  • 5
  • 17
6
votes
2 answers

Airflow: ExternalTaskSensor doesn't work as expected. Different task schedules

Colleagues, we need help. There are two dags Parent and Child, parent has its own schedule, suppose '30 * * * * ', child '1 8-17 * * 1-5', child waits for parent to execute, for example 40 minutes, if parent ends with error, then child also crashes…
Дмитрий Пырин
  • 61
  • 1
  • 3
6
votes
1 answer

How to set a number as retry condition in airflow DAG?

In my Airflow DAG i have 4 tasks task_1 >> [task_2,task_3]>> task_4 task_4 runs only after a successful run of both task_2 and task_3 How do i set a condition such as : if task_2 fails, retry task_2 after 2 minutes and stop retrying after the 5th…
user14123081
6
votes
1 answer

how to get the list of all the failed tasks from different dags

We have below command, we can clear the failed tasks and rerun them in a single attempt airflow clear [-s START_DATE] [-e END_DATE] --only_failed dag_id Is there any way we can get the information of all the failed tasks from all the dags and…
Ravi
  • 793
  • 3
  • 16
  • 29
6
votes
1 answer

Deleting airflow logs in scheduler folder

Currently i am using Airflow with Version : 1.10.10 After opening into airflow/logs folder there are many folder that are named based on your DAG name but there is a folder named scheduler which when opened consist folder that are named in date…
Anindhito Irmandharu
  • 135
  • 1
  • 8
6
votes
2 answers

Prevent Disabled DAG from Running Instantly When Enabled

In Airflow 1.10.10, when a disabled DAG is enabled by pressing the button to change it's state from Off to On, the DAG will always run once before the next scheduled run time. Is it possible to set a DAG to be enabled, but not make its first run…
Athena Wisdom
  • 6,101
  • 9
  • 36
  • 60
6
votes
1 answer

Architecturing Airflow DAG that needs contextual throttling

I have a group of job units (workers) that I want to run as a DAG Group1 has 10 workers and each worker does multiple table extracts from a DB. Note that each worker maps to a single DB instance and each worker needs to successfully deal with 100…
Rafay
  • 6,108
  • 11
  • 51
  • 71
6
votes
2 answers

How to get Airflow user who manually trigger a DAG?

In the Airflow UI, one of the log events available under "Browser > Logs" is the event "Trigger" along with the DAG ID and Owner/User who's responsible for triggering this event. Is this information easily obtainable programmatically? The use case…
ttechdo
  • 73
  • 1
  • 3
6
votes
0 answers

Airflow - Deleted task instance not re-running after being deleted from DAG search results page in UI

We are running a large number of DAGS, and as a result, thousands of task instances for each calendar date. We have sporadic days that have failed and we are attempting to re-run those dates as efficiently as possible. The method we attempted to…
phenderbender
  • 625
  • 2
  • 8
  • 18
6
votes
1 answer

Airflow DAG Scheduling for end of month

I want to run a schedule on Airflow (v1.9.0). My DAG needs to run at every end of month, but I don't know how to write the settings. my_dag = DAG(dag_id=DAG_ID, catchup=False, default_args=default_args, …
manomechi
  • 63
  • 1
  • 3
6
votes
3 answers

How to set priority across different DAGs in Airflow

Let's say we have two DAGs, dag1 and dag2, they serve different business requirements. they are completely unrelated. but dag1 is more important to have it finished as early as possible. For simplicty, they both have only one task and they run…
Fadi Bakoura
  • 301
  • 4
  • 14
1 2 3
…
83 84