Airflow 2: I have pushed an xcom from taskA and I am pulling that xcom within subdag taskB. I have been unable to pull the necessary xcom. I was wondering how one would do this.
I was able to pull the necessary xcom with same piece of code in Airflow 1.10(additionally I was passing provide_context=True, is depricated in Airflow 2), but its failing in Airflow 2 with error message as:
jinja2.exceptions.UndefinedError: 'DAG_ID' is undefined
Code Below :
DAG_ID = 'test_dag_name'
dag = DAG(
DAG_ID,
default_args=default_args, catchup=False,
max_active_runs=1, schedule_interval=SCHEDULE_INTERVAL
)
with dag:
test_bq = SubDagOperator(
task_id='test_task_id',
subdag=subdag_func(parent_dag_name=DAG_ID,
child_dag_name='test_task_id',
args=default_args,
run_date=run_date,
max_dt="{{ ti.xcom_pull(dag_id=DAG_ID ,task_ids='task_id_test')[0][0]}}"))
Note : If the exact dag name is passed in place of DAG_ID in xcom_pull, it works fine. But when passing the param(dag_id=DAG_ID),it fails.
Also, if xcom_pull(dag_id=None) as dag_id is optional from source code link, new error pops up as :
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1166, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1248, in _prepare_and_execute_task_with_callbacks
self.render_templates(context=context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1761, in render_templates
self.task.render_template_fields(context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 997, in render_template_fields
self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1010, in _do_render_template_fields
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1047, in render_template
return jinja_env.from_string(content).render(**context)
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "<template>", line 68, in top-level template code
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/sandbox.py", line 384, in getitem
return obj[argument]
jinja2.exceptions.UndefinedError: None has no element 0```
