Following this Stack Overflow answer, here's a toy example that executes a callback to a Celery chord whether or not the header was successful.
tasks.py
:
from celery import group, chord, Celery
cel = Celery(__name__,
broker='redis://localhost:6379',
backend='redis://localhost:6379')
@cel.task
def add(a, b):
print("adding {} + {}".format(a, b))
return a + b
@cel.task
def mult(a, b):
print("multiplying {} * {}".format(a, b))
return a * b
@cel.task
def div(a, b):
print("dividing {} by {}".format(a, b))
return a / b
@cel.task
def subtract(a, b):
print("subtracting {} from {}".format(b, a))
return a - b
@cel.task
def div_with_err(a, b):
print("This task was called with arguments {} and {} and should raise an error.".format(a, b))
return 1/0
@cel.task(name='tasks.callback')
def callback(*args, **kwargs):
print("Callback is executing")
print("args are ", args)
print("kwargs are ", kwargs)
group_foo = group([subtract.s(1, 3), add.s(2, 4)])
chain_bar = mult.s(1, 3) | div.s(4)
chain_with_err = mult.s(1, 3) | div_with_err.s(4)
header = group([group_foo, chain_with_err])
callback_baz = callback.s()
callback_baz.set(link_error=['tasks.callback'])
job = chord(header, callback_baz)
job.apply_async()
But I'm stuck on what I really want, which is to have the callback be a chain that executes regardless of whether the chord met success or failure. Here's something that I tried very naively, not really expecting it to work:
@cel.task(name='tasks.callback')
def callback_task_1(*args, **kwargs):
print("Callback is executing")
print("args are ", args)
print("kwargs are ", kwargs)
return 1 + 1
@cel.task
def callback_task_2(a):
print("Callback task 2 is executing; received {} from callback task 1".format(a))
callback_chain = chain(callback_task_1.s(), callback_task_2.s(), name='callback_chain')
group_foo = group([subtract.s(1, 3), add.s(2, 4)])
chain_bar = mult.s(1, 3) | div.s(4)
chain_with_err = mult.s(1, 3) | div_with_err.s(4)
header = group([group_foo, chain_with_err])
callback_baz = callback_task_1.s()
callback_chain.set(link_error=['callback_chain'])
job = chord(header, callback_chain)
job.apply_async()
and it didn't. It fails with
[2020-05-08 13:21:15,480: ERROR/MainProcess] Received unregistered task of type 'callback_chain'.
The message has been ignored and discarded.
Can anyone suggest another approach? If worse comes to worse in my real application I could make the functions represented by callback_chain
one big task, but that would really be a bummer. For instance, the last task in the chain is a send_email
task that's used many places in the application, and I don't want to duplicate all its functionality in a dedicated task just for this occasion.