7

I'm integrating a 3rd party C++ package to a python application using SWIG. The package connects to a proprietary API over a network and receives updates. The overall flow is that python instantiates a C++ object, calls its functions to set it up and then waits for the updates.

I implemented a callback mechanism for the updates using SWIG's directors feature, and while testing from python, or from C++ functions called by python, it works well. Namely I'm able to inherit a C++ class in Python, call its virtual functions from C++ and see the python code take priority and execute.

The problem:
When I receive the updates from the network I get:

The thread 'Win32 Thread' (0x1f78) has exited with code 0 (0x0).
Unhandled exception at 0x1e0650cb in python.exe: 0xC0000005: Access violation writing location 0x0000000c.

This exception is thrown from within python27.dll while calling the callback function.
My suspicion is this: I've violated the GIL
AFAIU the updates come from a different thread and call python's code using that thread.

At this point I'm at a loss. Is SWIG's director feature limited only to flows initiated within python (i.e. from python managed threads)?
How do I circumvent this? How do I induce updates from C++ to python? Is it even possible using SWIG?
Should I use a completely different approach?

I'm open for any suggestions on this matter...

Jonathan Livni
  • 101,334
  • 104
  • 266
  • 359
  • [This post on SWIG mailing list](http://article.gmane.org/gmane.comp.programming.swig/5306/match=multi+threading+python) talks exactly about this problem and calls for action regarding fixing it in SWIG. I guess this has not been dealt with since 2004 – Jonathan Livni Mar 06 '12 at 07:09
  • You can pass `-thread` to swig, and it will generate code to acquire the GIL whenever your python callback is invoked (i.e. the callback via swig's director feature). Here's a good discussion on the topic: https://github.com/swig/swig/issues/927#issuecomment-289279243 – Ryan Henning Jan 22 '21 at 00:41

1 Answers1

6

If your SWIG-wrapped C++ code invokes the callback routine in-thread, then there're probably no GIL problems - SWIG-generated code does not perform any GIL management that I've seen, which means that when Python code calls in to your C++ code, you retain the GIL throughout the call.

However, if your C++ code defers the callback to another thread, then you very likely have violated the GIL. This is simple enough to work around: Before you invoke the callback, call PyGILState_Ensure(), and when the callback completes, invoke PyGILState_Release. Refer to http://docs.python.org/c-api/init.html, the section "Non-Python created threads". (If you're using C++ exception handling here, you might need to take extra care to ensure that you can release the GIL.)

If you haven't yet looked at the stack trace, it's worth verifying that that NULL pointer deref isn't something silly happening in your code. (You can attach to the Python process running your code with VS/GDB/WinDBG; the Python execution will remain inscrutable, but you can trace your C++ code this way.)

lyngvi
  • 1,312
  • 12
  • 19
  • I found that if you're using SWIG directors, then you will need to lock the GIL yourself if another thread/process will trigger the callback. – AndyG Oct 14 '16 at 18:39