How to get conf value from airflow dag?

Question

I want to get conf value from dag area.

"{{ dag_run.conf['company'] }}" is recognized as a string.

How can I get this value?

Values are passed fine when calling other dags.

t_trigger = TriggerDagRunOperator(
    task_id="t-trigger",
    trigger_dag_id="example_dag",
    conf={
        "company": "{{ dag_run.conf['company'] }}",
    },
)

However, in the dag area the value is recognized as a string.

t_task_a = PythonOperator(
    task_id="t-task-a",
    python_callable=task-a,
)
employees = Variable.get(
    "{{ dag_run.conf['company'] }}", # problem
    default_var=['company'],
    deserialize_json=True
)
for employee in employees:
    t_employee_operator = PythonOperator(
        task_id=f"t-test-operator",
        python_callable=employee_operator,
        op_kwargs={"employee": employee}
    )
    t_task_a >> t_employee_operator

You should not use `Variable.get()` in top level code as this is very abusive. See https://stackoverflow.com/a/68257891/14624409 for reasoning why. — Elad Kalif, May 07 '22 at 06:51
As for templating. You can not template outside of operators templated fields. See https://stackoverflow.com/a/72137494/14624409 — Elad Kalif, May 07 '22 at 06:53

score 1 · Answer 1 · answered May 08 '22 at 07:20

As you already noticed Airflow does not render templates outside of Operators scope - this is expected. You can read this answer for more information about it.

Furthermore you are using Variable.get() as top level code which is very abusive. I explained the reasons in detail in this answer.

I would suggest to redesign your DAG structure to fit Airflow DAG writing practices.

How to get conf value from airflow dag?

1 Answers1

Linked