Airflow default on_failure_callback

Question

In my DAG file, I have define a on_failure_callback() function to post a Slack in case of failure.

It works well if I specify for each operator in my DAG : on_failure_callback=on_failure_callback()

Is there a way to automate (via default_args for instance, or via my DAG object) the dispatch to all of my operators?

interesting question, the on_failure_callback was defined at BaseOperator, the only way I can think of is to create your own operator and inherit from BaseOperator, then pass your on_failure_callback() there. Would like to see how other people think — Chengzhi, Jul 25 '17 at 20:29
Thanks for your opinion, but I wasn't confident about changing something as elementary as the BaseOperator. I prefer adding it manually to each operator but not to miss an update of the BaseOperator (less maintenance) — Pierre CORBEL, Jul 27 '17 at 06:12
did you encounter this error? https://stackoverflow.com/questions/50227670/apache-airflow-cannot-load-the-dag-bag-to-handle-failure — Lin Forest, May 08 '18 at 07:01

score 29 · Accepted Answer · answered Sep 19 '17 at 07:46

I finally found a way to do that.

You can pass your on_failure_callback as a default_args

class Foo:
  @staticmethod
  def get_default_args():
      """
      Return default args
      :return: default_args
      """

      default_args = {
          'on_failure_callback': Foo.on_failure_callback
      }

      return default_args

  @staticmethod
  def on_failure_callback(context):
     """
     Define the callback to post on Slack if a failure is detected in the Workflow
     :return: operator.execute
     """

     operator = SlackAPIPostOperator(
         task_id='failure',
         text=str(context['task_instance']),
         token=Variable.get("slack_access_token"),
         channel=Variable.get("slack_channel")
     )

     return operator.execute(context=context)

How is this used? Is this a subclass of the DAG class? – postelrich Jun 28 '19 at 15:12 — postelrich, Jun 28 '19 at 15:12
This does not answer the question. – Dommondke May 15 '23 at 15:13 — Dommondke, May 15 '23 at 15:13

score 2 · Answer 2 · answered Jan 05 '22 at 16:35

Late answer here, but yes you can specify an on_failure_callback in defaults for the DAG. You just have to write a custom function, making sure it can take in the context. Example:

def failure_callback(context):
    message = [
        ":red_circle: Task failed",
        f"*Dag*: {context['dag_run'].dag_id}",
        f"*Run*: {context['dag_run'].run_id}",
        f"*Task*: <{context.get('task_instance').log_url}|*{context.get('task_instance').task_id}* failed for execution {context.get('execution_date')}>",
    ]

    # Replace this return with whatever you want
    # I usually send a Slack notification here
    return "\n".join(message)


with DAG(
    ...
    default_args={
        ...
        "on_failure_callback": failure_callback,
    },
) as dag:
    ...

Airflow default on_failure_callback

2 Answers2

Linked