8

I am working on a DAG which depends on another DAG. Therefore I am using an ExternalTaskSensor. However, when working with this Sensor, I noticed some odd behavior;

If you set the soft_fail parameter to True (if it fails a task, it will set the state to skipped instead of failed), the task will never retry. While I would expect the task to retry as many times as specified (through the retries parameter). If you put the soft_fail parameter to False, it does retry. See a minimal example DAG below.

Am I missing something? 

from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.sensors import ExternalTaskSensor
from datetime import datetime, timedelta


dag_name = 'soft_fail_example'
schedule_interval = "0 * * * *"
default_args = {
            'owner': 'airflow',
            'depends_on_past': False,
            'start_date': datetime(2018, 1, 1),
            'email': [],
            'email_on_failure': False,
            'email_on_retry': False,
            'retries': 1,
            'retry_delay': timedelta(seconds=30),
        }

test_dag = DAG(dag_name, default_args=default_args, schedule_interval=schedule_interval, 
catchup=False, max_active_runs=1)


ets = ExternalTaskSensor(task_id="test_external_task_sensor", dag=test_dag, soft_fail=True, 
timeout=10, retries=5, poke_interval=1, external_dag_id="dependent_dag_id", 
external_task_id="dependent_task_id")

dummy_task = DummyOperator(task_id="collection_task", dag=test_dag)

dummy_task << ets
J.Brouwers
  • 411
  • 4
  • 13
  • 1
    I don't think you are missing anything. I would guess that tasks that fall into `skipped` state are just not retried. `soft_fail` is one way to get to the `skipped` state. I suggest you post this as to the airflow JIRA as a bug, as I think you are correct that the expected behavior should be to retry. – 7yl4r Jun 22 '18 at 15:33

0 Answers0