0

I have a DAG which has 5 parallel task. I don't want to execute them all in one go. I want to pass some flag/value to the task suppose if the flag is set to True it runs and if the flag is set to False it gets skip. Can we do this in Airflow2. The default arguments are very basic one and I have not scheduled the DAG.

My DAG Flow looks something like this

starttask>>5paralleltask>>endtask

And these 5 parallel task I have created using a for loop

Thank you so much in advance

Jaishree Rout
  • 382
  • 5
  • 17

2 Answers2

0

I'm not sure that i understand the requirement. If you want to execute all tasks but to limit the concurrency then use max_active_tasks: the number of task instances allowed to run concurrently

So in your case you need to set:

from airflow import DAG
with DAG(
    dag_id='somedag',
    ...,
    max_active_tasks=1,
) as dag:
    ...

If you want to add skip logic thus in some cases a task will run but in other it won't then you need to add operators like: ShortCircuitOperator, BranchPythonOperator etc that will decide when a task should be skipped.

Elad Kalif
  • 14,110
  • 2
  • 17
  • 49
  • Thanks Elad for your help I was more looking into run selected task at a time and to skip few task task at the same time. I apologies if y question is not very clear. I figured it out how to do it. Adding the answer so other can take the benefit – Jaishree Rout Mar 30 '22 at 13:52
0

The 5 parallel task that I have are bash operators. To skip few task and to run few task I forced the selected task to fail and to show is as skipped I used exit 99 with my bash command like:

'echo "running";exit 99'

There are two steps:

  • First to force the task to fail.
  • Second is to skip the task if it fails

To force fail the task I have added a list of 5 flags(True/False) if the flag is True the task runs if It is false it get forced failed and then skipped. for skipping the task I used exit 99.

And it is working as I expected in my case.

Jaishree Rout
  • 382
  • 5
  • 17