Does Airflow automatically detect changed variables that are used by already-deployed DAGs, and apply the change immediately, or is a DAG manual restart or refresh required to apply the new value of a changed variable?
Asked
Active
Viewed 841 times
2 Answers
1
Airflow Variables are stored in the database. Airflow does not maintain DAG <-> Variable relationships. Variables are not bound to a specific DAG.
The value of a variable is populated when Variable.get() is called in your code.
Elad Kalif
- 14,110
- 2
- 17
- 49
1
If the airflow variable is being used in the DAG code, something like the below, the change (to the python variable interval) will be set to the value of that airflow variable when the scheduler is executed next time. The scheduler executes periodically with a short interval.
# experimental_dag.py
default_args = {
'owner': 'anonymous',
'start_date': datetime.datetime(2021, 12, 1),
}
interval = Variable.get('interval', '@daily')
dag = DAG(
'experimental_dag',
default_args=default_args,
schedule_interval=interval
)
def write_log(ts):
with open('./output.txt', 'a') as f:
f.write(f'{ts}\n')
py_task = PythonOperator(
task_id='load_yesterday_data',
python_callable=write_log,
op_kwargs={
'ts': '{{ ts }}'
},
dag=dag
)
If the DAG code is changed, for example the above experimental_dag.py, in this situation, definitely, the DAG needs to be copied to the dag folder (configured in airflow.cfg).
sadcat
- 46
- 5
-
1Setting `Variable.get()` as top level code will cause serious problems with the database. Don't do that. – Elad Kalif Dec 09 '21 at 14:20
-
@Elad can you elaborate? What sort of problems, and why? – starmandeluxe Dec 10 '21 at 08:17
-
@starmandeluxe See my answer on https://stackoverflow.com/a/68257891/14624409 – Elad Kalif Dec 10 '21 at 10:54
-
@Elad thanks. It seems like the answer itself is solid other than the variable retrieval which there is a workaround for. – starmandeluxe Dec 11 '21 at 19:15