5

At first working with dag callback (on_failure_callback and on_success_callback), I thought it would trigger the success or fail statuses when the dag finishes (as it is defined in dag). But then it seems to be instanciated at every task instance and not dag run, so if a DAG has N tasks, it will trigger these callbacks N times.

I'm trying to catch the task-id and so send to slack. Reading another related question I came up with the below:

def success_msg(context):
    slack.slack_message(context['task_instance']); #send task-id to slack

def failure_msg(context):
    slack.slack_message(context['task_instance']); #send task-id to slack

default_args = {
    [...]
    'on_failure_callback': failure_msg,
    'on_success_callback': success_msg,
    [...]
}

But it fails, how should I parse the context variables and so be allowed to get the task-id?

Julinho da Adelaide
  • 386
  • 2
  • 5
  • 15
  • As stated here https://stackoverflow.com/questions/51147762/airflow-proper-way-to-handle-dags-callbacks callbacks work only on task level in 1.9.0 I think in 1.10.0 the DAG can also be a callback level. – tobi6 Jul 04 '18 at 06:45
  • yes its calling callback function at DAG level but I am not able to access context in that callback function I am also doing same thing written above \ any help? – Akshay Lande Jun 22 '20 at 11:12

1 Answers1

15

You can access the task with the task object from within the context.

context['task'] should be the appropriate way to do this. To get the task name, use task_id:

context['task'].task_id

To find more objects available in the context, you can walk through the list here: https://airflow.apache.org/docs/apache-airflow/stable/macros-ref.html

irondwarf
  • 195
  • 1
  • 8
tobi6
  • 8,033
  • 6
  • 26
  • 41
  • 1
    It really worked. Thanks for the answer and the docs reference. – Julinho da Adelaide Jul 05 '18 at 05:49
  • how do you do to debug its items? I'm curious about getting the items from dag and task instance. I'd print them, but when I run like python dag.py it doesn't log... – Julinho da Adelaide Jul 12 '18 at 23:41
  • Please open a new stack overflow question for this. Also include examples since I am unsure what you are asking. – tobi6 Jul 13 '18 at 06:14
  • I tried to get context['task'] on both on_failure_callback for Task and DAG. For task, it has given its own task id but for DAG it has given task id for the last successful task instance. How can we get all failure task instances/IDs with their exceptions if possible in the on_failure_callback function for DAG? – Shubhank Gupta Nov 30 '22 at 12:02