I have this problem in a larger application, but have reduced it to the following MRE.
I have a Celery chord consisting of two parallel tasks feeding into one final tasks. It's expected that tasks might occasionally permanently fail for external reasons, simulated here by raising Exception. I wish to handle that error with a custom error handler, and not cause any unhandled execptions in the Celery worker process.
Here is the definition of my three example tasks in a file I call canvastest.py.
#!/usr/bin/env python
from celery import Celery
app = Celery('canvastest', backend='redis://localhost', broker='redis://localhost')
@app.task(throws=(Exception,),)
def t1(x):
print("t1")
return x
@app.task(throws=(Exception,),)
def t2(x):
print("t2")
return x
@app.task(throws=(Exception,),)
def t3(x):
print("t3")
return x
@app.task()
def error_handler(*args, **kwargs):
print("Error handler")
I build my chord as follows in the file main.py
from canvastest import t1, t2, t3, error_handler
from celery import chord
if __name__ == '__main__':
combined_task = chord((t1.s(1), t2.s(2)),t3.s()).on_error(error_handler.si())
combined_task()
To run it I first start a local redis instance by running docker run -p 6379:6379 redis then I run the worker celery -A canvastest worker and the main application python main.py
The log output of the worker then as expected looks as follows
[2023-04-20 13:59:00,840: WARNING/ForkPoolWorker-16] t1
[2023-04-20 13:59:00,842: WARNING/ForkPoolWorker-1] t2
[2023-04-20 13:59:00,854: WARNING/ForkPoolWorker-16] t3
If I modify t3 to fail by changing it as
@app.task(throws=(Exception,),)
def t3(x):
print("t3")
raise Exception()
return x
the logs correctly looks as follows
[2023-04-20 14:00:18,078: WARNING/ForkPoolWorker-16] t1
[2023-04-20 14:00:18,080: WARNING/ForkPoolWorker-1] t2
[2023-04-20 14:00:18,090: WARNING/ForkPoolWorker-16] t3
[2023-04-20 14:00:18,092: WARNING/ForkPoolWorker-16] Error handler
but if we move the error to t1 or t2 as
@app.task(throws=(Exception,),)
def t2(x):
print("t2")
raise Exception()
return x
then this happens
[2023-04-20 14:01:31,667: WARNING/ForkPoolWorker-16] t1
[2023-04-20 14:01:31,668: WARNING/ForkPoolWorker-1] t2
[2023-04-20 14:01:31,675: ERROR/ForkPoolWorker-1] Chord '3336718d-57e3-41e0-b62d-4fad8ae9cc62' raised: ChordError('Dependency ffdd942e-cfe9-498f-a4b1-9d3a80a9ee45 raised Exception()')
Traceback (most recent call last):
File "/home/jerkern/celery_mre/.venv/lib/python3.10/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/jerkern/celery_mre/.venv/lib/python3.10/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/home/jerkern/celery_mre/canvastest.py", line 15, in t2
raise Exception()
Exception
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jerkern/celery_mre/.venv/lib/python3.10/site-packages/celery/backends/redis.py", line 520, in on_chord_part_return
resl = [unpack(tup, decode) for tup in resl]
File "/home/jerkern/celery_mre/.venv/lib/python3.10/site-packages/celery/backends/redis.py", line 520, in <listcomp>
resl = [unpack(tup, decode) for tup in resl]
File "/home/jerkern/celery_mre/.venv/lib/python3.10/site-packages/celery/backends/redis.py", line 426, in _unpack_chord_result
raise ChordError(f'Dependency {tid} raised {retval!r}')
celery.exceptions.ChordError: Dependency ffdd942e-cfe9-498f-a4b1-9d3a80a9ee45 raised Exception()
[2023-04-20 14:01:31,676: WARNING/ForkPoolWorker-1] Error handler
There is now an unhandled exception and stack trace in the log. This is the problem, and I wish to get rid of that. The error has already been handled by the error_handler method, and I don't wish to pollute the logs by surplus exceptions and stack traces.
Is there a way to silence this, or is it a bug in celery? My expectation would be that the error_handler is invoked in the same way regardless of which of the three tasks raises the exception, and that not further exceptions are raised and logged as a side-effect of the task failing (since that is expected behavior).
