communication between top level python code and operator variable in airflow

Question

i am writing airflow code where i am running one python operator to read user input given from cli and want to use result return by that operator in top level python code for dynamic task generation . using Xcom we can perform inter task communication but how can we communicate between task and main code.

Answer 1

This is impossible (and I believe you had this answer from me several times in Airlfow Slack).

Task execution cannot influence DAG structure of Airflow. Full Stop.

We are working on partial implementation of Dynnamic Tasks Mapping to implement Map/Reduce kind of behaviour https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-42+Dynamic+Task+Mapping but this does not change the structure of the DAG or does not change Dag structure - it merely makes single task to be able to fan-out and then fan-in into multiple tasks.