Airflow start multiple concurrent generic tasks

Question

Trying to get a few tasks concurrently on cloud composer:

arr = {}
for i in xrange(3):
    print("i: " + str(i))
    command_formatted = command_template.format(str(i))
    create_training_instance = bash_operator.BashOperator(
        task_id='create_training_instance',
        bash_command=command_formatted)
    arr[i] = create_training_instance
    start_training.set_downstream(arr[i])

Getting the following error:

Broken DAG: [/home/airflow/gcs/dags/scale_simple.py] Dependency , create_training_instance already registered

score 1 · Answer 1 · answered Sep 04 '18 at 20:31

1

The task_id should always be unique for a single task. So, you can use something like create_training_instance_{}.format(i) as task_id.

answered Sep 04 '18 at 20:31

kaxil

17,706
2
59
78

score 0 · Accepted Answer · answered Sep 04 '18 at 16:51

0

You need to parameterize your task id as well, e.g., task_id='create_training_instance' --> 'create_traiing_instance-{}'.format(i)

answered Sep 04 '18 at 16:51

JY2k

2,879
1
31
60

Airflow start multiple concurrent generic tasks

2 Answers2