0

I have a DAG that's running a few snowflake operators, and in the SQL files I have templated dates as follows:

'{{ prev_execution_date.subtract(minutes=15).in_tz('America/Toronto').to_datetime_string() }}'
'{{ execution_date.in_tz('America/Toronto').to_datetime_string() }}'

This all works fine.

I also want to be able to manually trigger the DAG and pass in the date as well, so I tried the following in the the query file

{{ dag_run.conf['startdate'] if dag_run else prev_execution_date.subtract(minutes=15).in_tz('America/Toronto').to_datetime_string()  }}

(basically the solution in this question)

This works fine in the manually triggered instance where I pass in the startdate value, but the else clause always returns blank when it's a scheduled instance.

Am I missing something in the else clause or is there a different solution that bypasses the if statement altogether?

I'm on Airflow 1.10.12.

Thanks!

Andrew Cheng
  • 13
  • 2

1 Answers1

0

This is because the dag_run object always exists whether or not it is a manual run or scheduled run.

from datetime import datetime

from airflow.models import DAG
from airflow.operators.python_operator import PythonOperator

dag = DAG(
    dag_id="trigger_dag",
    start_date=datetime(2021, 4, 5),
    catchup=True,
    schedule_interval='@daily',
)
with dag:
    op = PythonOperator(
        task_id='a',
        python_callable=lambda x: print(x),
        op_kwargs={
            "value": "{{ dag_run.conf['startdate'] if dag_run['external_trigger'] else prev_execution_date.subtract(minutes=15).in_tz('America/Toronto').to_datetime_string() }}"
        },
    )

Here is the output with the modified execution date value.

[2021-04-08 05:46:53,430] {taskinstance.py:901} INFO - Executing <Task(PythonOperator): a> on 2021-04-07T00:00:00+00:00
[2021-04-08 05:46:53,434] {standard_task_runner.py:54} INFO - Started process 2604 to run task
[2021-04-08 05:46:53,462] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'trigger_dag', 'a', '2021-04-07T00:00:00+00:00', '--job_id', '35', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/trigger.py', '--cfg_path', '/tmp/tmpe8zsnj8x']
[2021-04-08 05:46:53,463] {standard_task_runner.py:78} INFO - Job 35: Subtask a
[2021-04-08 05:46:53,509] {logging_mixin.py:112} INFO - Running <TaskInstance: trigger_dag.a 2021-04-07T00:00:00+00:00 [running]> on host 953a6668d603
[2021-04-08 05:46:53,564] {logging_mixin.py:112} INFO - 2021-04-05 19:45:00
[2021-04-08 05:46:53,565] {python_operator.py:114} INFO - Done. Returned value was: None
[2021-04-08 05:46:53,574] {taskinstance.py:1070} INFO - Marking task as SUCCESS.dag_id=trigger_dag, task_id=a, execution_date=20210407T000000, start_date=20210408T054653, end_date=20210408T054653
[2021-04-08 05:46:58,385] {local_task_job.py:102} INFO - Task exited with return code 0

Here is the output with the value from conf passed from the manually triggered DAG.

[2021-04-08 05:47:47,431] {taskinstance.py:901} INFO - Executing <Task(PythonOperator): a> on 2021-04-08T05:47:31.333405+00:00
[2021-04-08 05:47:47,434] {standard_task_runner.py:54} INFO - Started process 2665 to run task
[2021-04-08 05:47:47,463] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'trigger_dag', 'a', '2021-04-08T05:47:31.333405+00:00', '--job_id', '36', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/trigger.py', '--cfg_path', '/tmp/tmpwoshw679']
[2021-04-08 05:47:47,464] {standard_task_runner.py:78} INFO - Job 36: Subtask a
[2021-04-08 05:47:47,512] {logging_mixin.py:112} INFO - Running <TaskInstance: trigger_dag.a 2021-04-08T05:47:31.333405+00:00 [running]> on host 953a6668d603
[2021-04-08 05:47:47,564] {logging_mixin.py:112} INFO - super-duper
[2021-04-08 05:47:47,564] {python_operator.py:114} INFO - Done. Returned value was: None
[2021-04-08 05:47:47,574] {taskinstance.py:1070} INFO - Marking task as SUCCESS.dag_id=trigger_dag, task_id=a, execution_date=20210408T054731, start_date=20210408T054747, end_date=20210408T054747
[2021-04-08 05:47:52,388] {local_task_job.py:102} INFO - Task exited with return code 0
Alan Ma
  • 501
  • 2
  • 7