Problem defining DAG with Airflow 2 Taskflow API

Question

I created a DAG with airflow 2 Taskflow API:

with airflow.DAG("plot", schedule_interval=None, default_args=default_args) as dag:
    cf = collect_files()
    upi = update_process_info(cf)
    for i in range(0, max_parallel_plot_tasks):
        plot_files(cf, i, int(max_parallel_plot_tasks)) >> upi

How I get rid of the connection of "collect_files" to "update_process_info" with the Taskflow API?

Graph:

Regards Oli

score 1 · Answer 1 · answered Feb 26 '21 at 08:14

Try something like:

with airflow.DAG("plot", schedule_interval=None, default_args=default_args) as dag:
    cf = collect_files()
    upi = None
    for i in range(0, max_parallel_plot_tasks):
        if not upi:
            upi = update_process_info(plot_files(cf, i, int(max_parallel_plot_tasks)))
        else:
            plot_files(cf, i, int(max_parallel_plot_tasks)) >> upi

Problem defining DAG with Airflow 2 Taskflow API

1 Answers1