xcom_pull is not working in dynamic task generation function

Question

I have a function which dynamically creates a sub task, where I am reading value from xcom_pull but there I am getting the error:

  File "/home/airflow/gcs/recon_nik_v6.py", line 168, in create_audit_task
    my_dict=kwargs["ti"].xcom_pull(task_ids='accept_input_from_cli', key='my_ip_dict')
KeyError: 'ti'

If I use the same my_dict=kwargs["ti"].xcom_pull(task_ids='accept_input_from_cli', key='my_ip_dict') code in another function then it works, but in this dynamic part it's not working.

you need to pass `**kwargs` in the function parameter. Please post your code too when you ask questions. — Jacob Celestine, Dec 30 '21 at 15:42

Jarek Potiuk · Accepted Answer · 2022-01-02T12:48:59.053

Ssimilarly to your other questions (and explained in slack several times). This is not how Airflow works.

XCom pull and task instances are only available when DAG Run is being executed. When you create your DAG structure (i.e. dynamically generate DAGs) you cannot use them.

Only Task Instances when executing tasks can access them and this is already long after the DAGs have been parsed and DAG structure established.

So what you try to do is simply impossible.

xcom_pull is not working in dynamic task generation function

1 Answers1