Run a chord callback even if the main tasks fail

Question

Is it possible to run a chord callback even if the main tasks failed?

I've created a chord which I added a bunch of tasks and registered a callback to it. My problem is that if one of the tasks fail, the callback is not triggered. But I would like the callback to be triggered either way.

I've tried to register the callback with si() (immutability)

callback = tasks.run_delete_rule.si([timestamp])
header = [tasks.run_update_rule.s(i, timestamp) for i in item_ids]
result = chord(header)(callback)

I also tried to add the parameter ignore_result=True to both tasks decorator, but no success.

I am looking for this same solution. – Nishant Kashyap Apr 13 '15 at 11:50 — Nishant Kashyap, Apr 13 '15 at 11:50

alejandrodnm · Accepted Answer · 2015-07-17T07:42:04.853

From the github issue #1881 if the callback has the link_error option set, which takes a list of tasks names, then when a task of the chord fails the link_error tasks are going to be executed.

@task(name='super_task.good')
def good():
    return True

@task(name='super_task.raise_exception')
def raise_exception():
    raise ValueError('error')

@task(name='super_task.callback')
def callback(*args, **kwargs):
    logger.info('callback')
    logger.info(args)
    logger.info(kwargs)
    return 'finished'

@task(name='super_task.error_callback')
def error_callback(*args, **kwargs):
    logger.info('error_callback')
    logger.info(args)
    logger.info(kwargs)
    return 'error'

>>> c = chord(
        [raise_exception.s(), good.s(), raise_exception.s()], 
        callback.s().set(link_error=['super_task.error_callback'])
    )
>>> result = c()

This will execute the chord and in your celery log, you'll see the raise_exception task fail, and the execution of error_callback which will receive in it's args the task_id of callback.

At this point the value of result will contain the AsyncResultinstance of callback, and because in a chord the errors propagate to the callback doing result.get() will raise the exception of the tasks and result.traceback gives you the traceback.

If you want to have a single callback, just pass the name of the chord callback to link_error

callback.s().set(link_error='super_task.callback')

NOTE

Another options it's to set CELERY_CHORD_PROPAGATES = False which will revert to the pre celery 3.1 behavior and always execute the callback.

But this is not a recommended approach because as you can find in the github issue #1349

Celery 3.1 defines how chord errors are handled, the previous behavior was never documented and more of an accident since it was never the intention to work that way.

We couldn't change the behavior in a bugfix release so a setting had to be used instead, but it was never the intention that someone would deliberately disable the new behavior.

The new behavior is there to protect against this sort of issue happening, and the backward compatible setting may be removed. I suggest you find some other way to handle errors here (and I wouldn't mind a proposal if you can invent a nice api for it)

How would I get the result of the good task in error_callback? — Carl, Nov 19 '15 at 01:19
@Carl I not 100% sure but I don't think you can. I think the only attributes you can get are the Exception and the Traceback of the task that failed. You can maybe use a data store if you need something specific to be retrieved later even if everything fails. We just needed it to send and email so I didn't play too much with that. — alejandrodnm, Nov 19 '15 at 08:07
According to the accept code, I get error callback run, but the chord function still raise exception, not return value from the error callback... — Wesley, Jul 10 '17 at 08:15
I don't think you should attach link_error to the callback. link_error should be attached to the chord (task) and so on. — Durai, Aug 17 '20 at 21:25

score 0 · Answer 2 · answered Jul 24 '19 at 13:34

You just have to change manner how link_error is called. Instead of string reference, pass signature with arguments you want.

In example above, you can pass argument as following

c = chord(
    [raise_exception.s(), good.s(), raise_exception.s()], 
    callback.s().set(link_error=[error_callback.s(<arguments_here>)])
)

Just keep on mind, first argument will be task_id and further arguments will be those defined in signature.

Run a chord callback even if the main tasks fail

2 Answers2

Linked