0

I have a question about the TriggerDagRunOperator , specifically the wait_for_completion parameter.

Before moving to Airflow 2.2, we used this operator to trigger another DAG and a ExternalTaskSensor to wait for its completion.

In Airflow 2.2, there is a new parameter that is called wait_for_completion that if sets to True, will make the task complete only when the triggered DAG completed.

This is great, but I was wondering about wether the worker will be released between pokes or not. I know that the ExternalTaskSensor used to have a parameter reschedule that you can use for pokes larger than 1m which will release the worker slot between pokes - but I don’t see it in the documentation anymore.

My question is if the wait_for_completion parameter causes the operator to release the worker between pokes or not? From looking at the code I don’t think that is the case, so I just want to verify.

enter image description here

If it isn’t releasing the worker and the triggered DAG is bound to take more than 1m to finish, what should be the best approach here?

We are using MWAA Airflow 2.2 so I guess deferred operators are not an option (if it is a solution in this case)

snir.isl
  • 155
  • 1
  • 12

1 Answers1

1

When using wait_for_completion=True in TriggerDagRunOperator the worker will not be released as long as the operator is running. You can see that in the operator implementation. The operator use time.sleep(self.poke_interval)

As you pointed there are two ways to achieve the goal of verifying the triggered dag completed:

  1. DAG A Using TriggerDagRunOperator followed by ExternalTaskSensor
  2. Using TriggerDagRunOperator with wait_for_completion=True

However other than resources issue which you mentioned the two options are not really equivalent.

In option 1 if the triggered DAG fails then the ExternalTaskSensor will fail.

In option 2 consider:

from airflow.operators.trigger_dagrun import TriggerDagRunOperator
my_op = TriggerDagRunOperator (
    task_id='task',
    trigger_dag_id="dag_b",
    ...,
    wait_for_completion=True,
    retries=2
)

if the dag_b fails then TriggerDagRunOperator will retry which will invoke another DagRun of dag_b.

Both options are valid. You need to decide which behavior suitable for your use case.

Elad Kalif
  • 14,110
  • 2
  • 17
  • 49
  • Thanks Elad for the answer. The only missing part is, how can I set a sensor/triggerDagRunOperator that will sleep between pokes, if possible, at MWAA 2.2.2? I know of deferrable operators, but they are not available right now in MWAA. – snir.isl Apr 06 '22 at 06:33
  • You can't. TriggerDagRunOperator is not a sensor. while sleeps it occupy the worker. – Elad Kalif Apr 06 '22 at 06:51
  • Is there a sensor that can release the worker slot between pokes then? – snir.isl Apr 06 '22 at 07:12
  • ah wait, I can see there is a mode parameter to the base sensor, so I can use the ExternalTaskSensor with reschedule mode to make the sensor sleep between pokes, right? – snir.isl Apr 06 '22 at 07:13
  • 1
    `ExternalTaskSensor` is sensor so yes but your question was on `TriggerDagRunOperator`. The `TriggerDagRunOperator` can not release worker, `ExternalTaskSensor` can just like all sensors can. – Elad Kalif Apr 06 '22 at 07:21
  • So, `ExternalTaskSensor` release the worker? I thought that between pokes, all sensors keeps running in the worker, hence keep it busy. Do you have any example? – Santiago P May 10 '22 at 11:24
  • 1
    @SantiagoP Sensors has `mode` parameter which can be configured. You can read about the differences between `poke` and `reschedule` modes: https://github.com/apache/airflow/blob/d7b85d9a0a09fd7b287ec928d3b68c38481b0225/airflow/sensors/base.py#L88-L99 – Elad Kalif May 10 '22 at 11:57