Airflow - Proper way to handle DAGs callbacks

Question

I have a DAG and then whenever it success or fails, I want it to trigger a method which posts to Slack.

My DAG args is like below:

default_args = {
    [...]
    'on_failure_callback': slack.slack_message(sad_message),
    'on_success_callback': slack.slack_message(happy_message),
    [...]
}

And the DAG definition itself:

dag = DAG(
    dag_id = dag_name_id,
    default_args=default_args,
    description='load data from mysql to S3',
    schedule_interval='*/10 * * * *',
    catchup=False
      )

But when I check Slack there is more than 100 message each minute, as if is evaluating at each scheduler heartbeat and for every log it did runned the success and failure method as if it worked and didn't work for the same task instance (not fine).

How should I properly use the on_failure_callback and on_success_callback to handle dags statuses and call a custom method?

Duplicate of this https://stackoverflow.com/questions/44586356/airflow-failed-slack-message? Rather than using `on_failure_callback` and `on_success_callback`, why not just make the slack message a task in your DAG as you are requesting a message whether the task is a success/failure. — Zack, Jul 03 '18 at 14:01
Not a duplicate, this question is specifically about usage of success/failure callbacks — cwurtz, Jul 03 '18 at 14:36

score 9 · Answer 1 · answered Jul 03 '18 at 14:31

The reason it's creating the messages is because when you are defining your default_args, you are executing the functions. You need to just pass the function definition without executing it.

Since the function has an argument, it'll get a little trickier. You can either define two partial functions or define two wrapper functions.

So you can either do:

from functools import partial

success_msg = partial(slack.slack_message, happy_message);
failure_msg = partial(slack.slack_message, sad_message);

default_args = {
    [...]
    'on_failure_callback': failure_msg
    'on_success_callback': success_msg
    [...]
}

or

def success_msg():
    slack.slack_message(happy_message);

def failure_msg():
    slack.slack_message(sad_message);

default_args = {
    [...]
    'on_failure_callback': failure_msg
    'on_success_callback': success_msg
    [...]
}

In either method, note how just the function definition failure_msg and success_msg are being passed, not the result they give when executed.

It did worked. But seems to spawn a message to each task and not to each dag run. My code right now is made to run at each dag run, I'm thinking of adapting it to each task run but then I need to retrieve its `task_id`, is that possible? Other way I think is to make it once per dag run so I don't need to change the code. Is this possible aswell? Many Thanks. — Julinho da Adelaide, Jul 03 '18 at 17:37
@JulinhodaAdelaide I think 1.9.0 can only define this on per task level, 1.10.0 will have a DAG based definition available. — tobi6, Jul 03 '18 at 18:25

score 5 · Answer 2 · answered May 14 '20 at 05:32

5

default_args expands at task level, therefore it becomes per task callback

apply the attribute at DAG flag level outside of "default_args"

answered May 14 '20 at 05:32

the pillow

412
6
14

The callback params does not work outside of the default_args. It has to be in the default_args which as mentioned works at task level. – Dhruv Kadia Oct 27 '22 at 19:42

score 0 · Answer 3 · answered Jul 03 '18 at 14:15

What is the slack method you are referring to? The scheduler is parsing your DAG file every heartbeat, so if the slack some function defined in your code, it is going to get run every heartbeat.

A few things you can try:

Define the functions you want to call as PythonOperators and then call them at the task level instead of at the DAG level.
You could also use TriggerRules to set tasks downstream of your ETL task that will trigger based on failure or success of the parent task.

From the docs: defines the rule by which dependencies are applied for the task to get triggered. Options are: { all_success | all_failed | all_done | one_success | one_failed | dummy}

You can find an example of how this would look here (full disclosure - I'm the author).

Airflow - Proper way to handle DAGs callbacks

3 Answers3

Linked