how to use up_for_reschedule with python operator

Question

in our orchestration we are using one DAG to trigger other child dags and till these child dags not finish our master DAG is running to check the status. we used sleep of 5 min to check status of child dags after each 5 min. As this task is continually in running state so its consuming resources on one worker. recently we came to know about up_for_reschedule. does this solve my problem to release the worker ? is it possible to use up_for_reschedule with python operator ? if yes is there any document which i can refer ?

Answer 1

There are a couple of options here.

1. You can use PythonSensor instead of Operator in reschedule mode. In reschedule mode, the airflow will reschedule the task instance if the sensor con

          task = PythonSensor(
                task_id='sensor_example',
                mode='reschedule',
                python_callable=func
            )

2. You can link a bunch of TriggerDagRunOperoator. But this is best if you just want to create the dog runs and not wait for status checks. Unlike sensors, it doesn't have a reschedule mode. So when wait_for_completion is True, it will hold the worker slot