2

Following this Stack Overflow answer, here's a toy example that executes a callback to a Celery chord whether or not the header was successful.

tasks.py:

from celery import group, chord, Celery

cel = Celery(__name__,
                broker='redis://localhost:6379',
                backend='redis://localhost:6379')

@cel.task
def add(a, b):
    print("adding {} + {}".format(a, b))
    return a + b

@cel.task
def mult(a, b):
    print("multiplying {} * {}".format(a, b))
    return a * b

@cel.task
def div(a, b):
    print("dividing {} by {}".format(a, b))
    return a / b

@cel.task
def subtract(a, b):
    print("subtracting {} from {}".format(b, a))
    return a - b

@cel.task
def div_with_err(a, b):
    print("This task was called with arguments {} and {} and should raise an error.".format(a, b))
    return 1/0

@cel.task(name='tasks.callback')
def callback(*args, **kwargs):
    print("Callback is executing")
    print("args are ", args)
    print("kwargs are ", kwargs)


group_foo = group([subtract.s(1, 3), add.s(2, 4)])

chain_bar = mult.s(1, 3) | div.s(4)

chain_with_err = mult.s(1, 3) | div_with_err.s(4)

header = group([group_foo, chain_with_err])

callback_baz = callback.s()

callback_baz.set(link_error=['tasks.callback'])

job = chord(header, callback_baz)

job.apply_async()

But I'm stuck on what I really want, which is to have the callback be a chain that executes regardless of whether the chord met success or failure. Here's something that I tried very naively, not really expecting it to work:

@cel.task(name='tasks.callback')
def callback_task_1(*args, **kwargs):
    print("Callback is executing")
    print("args are ", args)
    print("kwargs are ", kwargs)
    return 1 + 1

@cel.task
def callback_task_2(a):
    print("Callback task 2 is executing; received {} from callback task 1".format(a))

callback_chain = chain(callback_task_1.s(), callback_task_2.s(), name='callback_chain')

group_foo = group([subtract.s(1, 3), add.s(2, 4)])

chain_bar = mult.s(1, 3) | div.s(4)

chain_with_err = mult.s(1, 3) | div_with_err.s(4)

header = group([group_foo, chain_with_err])

callback_baz = callback_task_1.s()

callback_chain.set(link_error=['callback_chain'])

job = chord(header, callback_chain)

job.apply_async()

and it didn't. It fails with

[2020-05-08 13:21:15,480: ERROR/MainProcess] Received unregistered task of type 'callback_chain'.
The message has been ignored and discarded.

Can anyone suggest another approach? If worse comes to worse in my real application I could make the functions represented by callback_chain one big task, but that would really be a bummer. For instance, the last task in the chain is a send_email task that's used many places in the application, and I don't want to duplicate all its functionality in a dedicated task just for this occasion.

Katie
  • 808
  • 1
  • 11
  • 28
  • Did you try to run it without the `callback_chain.set(link_error=['callback_chain'])`? – DejanLekic May 08 '20 at 19:12
  • @DejanLekic when I comment out that line (and also the `chain_with_err` line), I get an interesting error: `send_task() got multiple values for argument 'name'`. And then, during handling of that exception, `TypeError: sequence item 1: expected a bytes-like object, NoneType found`. – Katie May 08 '20 at 20:38

0 Answers0