0

I have a requirement where I need the dag triggered by TriggerDagRunOperator to execute a backfill and not just for the same execution date.

The TriggerDagOperator is set as follows:

trigger1 = TriggerDagRunOperator(
  task_id = 'trigger1',
  trigger_dag_id = 'target_dag',
  conf = {'message': 'Starting target 1'},
  reset_dag_run = True,
  wait_for_completion = True
)

Target dag is basically:

starting_date = datetime.strptime("2021-11-15", "%Y-%m-%d")

with DAG("target_dag", default_args=default_args, schedule_interval='@daily', max_active_runs=10) as dag:

  start = DummyOperator(
    task_id = 'start'
  )

  t1 = PythonOperator(
    task_id = "t1",
    provide_context=True,
    python_callable=t1
  )

  finish = DummyOperator(
    task_id = 'finish'
  )

  start >> t1 >> finish

target_dag is only executing for today's date and not backfilling.

How do I force it to backfill regardless of past dag runs? I'm using airflow 2.0

Tessa Altman
  • 33
  • 4
  • What do you mean "backfill"? How this should work? – vitooh Nov 24 '21 at 16:02
  • @vitooh The target dag should run from start_date till today like a normal backfill run. But when triggered from another dag, it only runs for one day, the same day it was triggered. – Tessa Altman Nov 24 '21 at 17:35

1 Answers1

0

this might be late now, but I have come up with 2 different solutions.

The first one (and probably the better) would be as follows:

from airflow.operators.latest_only_operator import LatestOnlyOperator
t1 = LatestOnlyOperator(task_id="ensure_backfill_complete")

I was stuck on a similar conundrum, and this suddenly popped in my head.

The 2nd one is basically wrapping the operator in a loop within a python function, which is honestly terrible.

and it seems to work.

Islam Elsayed
  • 21
  • 1
  • 4