0

I have a python module with c++ code using pybind11 that I have imported in my django + celery 3.1.25 app. The c++ code contains assertions that might trigger inside of a celery worker which then leads to the WorkerLostError.

I have tried to put the calls to the python c++ module inside try: except: scopes, however this does not work and the celery worker still crashes.

I have also tried to bind an error callback function using the link_error= argument in my apply_async call to my celery task, however the method never gets called.

Ideally I would like to catch the error somehow so I can display an error message to the user. Any suggestions is appreciated!

Ronny
  • 454
  • 4
  • 15
  • Just to clarify: is the C++ module your own? Can you modify its source code? Can you compile it from source? – Alessandro Power Jan 09 '19 at 17:32
  • yes, I can modify the c++ source, but I put the asserts there to catch unexpected errors. – Ronny Jan 10 '19 at 18:36
  • The use of assertions should be restricted to debugging. For your production build I would compile with assertions disabled. You may want to throw an [exception](https://pybind11.readthedocs.io/en/stable/advanced/exceptions.html) instead of using an assert, but in general you should save asserts for checking your program's internal logic and use exceptions to signal the violation of a contract. – Alessandro Power Jan 10 '19 at 23:22
  • Yes, that would make sense, but these assertions are indeed for the internal logic of the program, like writing out of bounds of arrays etc. – Ronny Jan 13 '19 at 18:04
  • They should still be enabled. By all means, keep the assertions for unit testing and development builds. But in production, they should be disabled. – Alessandro Power Jan 13 '19 at 18:46
  • Sure but that is beside the problem, I don't see how that would be different if the program crashes due to a segfault for instance, I still am not able to handle the error – Ronny Jan 14 '19 at 19:10
  • If your code segfaults, you have no choice but to try catch the `WorkerLostError`. Ditto for failed assertions. – Alessandro Power Jan 14 '19 at 21:10

1 Answers1

0

What I usually do to handle the execution of C++ binaries from a Celery task is to use the subprocess module, in particular subprocess.run. You will get the result code (rc) and the standard output/error produced by the executed command without making the worker process crash (WorkerLost). At most, by calling subprocess.run you will have to take care of exceptions, that are raised upon non-zero rc's depending on the value of the keyword argument check:

If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured.

However, I usually prefer to disable this and check the return codes by myself.

I have used this approach extensively to handle both expected and unexpected return codes, as well as OS signals (rc < 0) produced by memory leaks, segfaults, etc.

To handle and format OS signals I use the signal module.

user17095708
  • 1
  • 1