How to dynamically organize group of operators of different lengths

Question

I am dynamically generating operators that end up filling a structure of operators like this:

operators_dict = {0: [snowflake_1, [mysql_1, s3_1]], 1: [snowflake_2, s3_2]}
interlude_operators = [dummy_op_1]
sfn_levels = 2

The operators_dict can have any number of keys, the values can have any number of items and they can be lists with 2 operators or single operators. It can happen that an specific operators_dict has only single operators like {0: [snow_1, snow_2, snow_3]...}, so the condition of having a nested list is not happening in all cases.

The desired output graph view for the DAG using the example is the following:

I have tried with this:

for i in range(sfn_levels - 1):
    operators_dict[i] >> interlude_operators[i] >> operators_dict[i + 1]

But I get the error

Relationships can only be set between Operators; received list

How could I solve the problem? Thank you very much in advance

You could use `TaskGroup`s and create the dependencies between them. Check the example in [this answer](https://stackoverflow.com/a/68367685/10569220). — NicoE, Jul 21 '21 at 11:54

score 1 · Accepted Answer · answered Jul 21 '21 at 14:54

You cannot do it this way. The >> operator takes only Task or List[Task] as left-hand operand. and You are passing it List[Union[Task, List[Task]].

You have to do a bit more manual work there and manually add some of the dependencies if you want to keep this structure.

The Task Group suggested by @NicoE is also good idea, then you could have arbitrary complex structure not limiting to tasks and list of tasks.

One more comments: updating the << operator to also work in this way and handle List[Union[Task, List[Task]] would be an interesting addition - maybe you could add a PR adding this feature to Airflow?

How to dynamically organize group of operators of different lengths

1 Answers1