0

I have C++ dll which works with multiple threads. So I wrapped this library with Cython and created special receiver callback-function, which must adds some results to asyncio.Queue.

cdef void __cdecl NewMessage(char* message) nogil:

I marked it as nogil, this callback calls from another thread. In this callback I just use:

with gil:
   print("hello")  # instead adding to Queue. print("hello") is more simple operation to demostrate problem

And got deadlock here. How to resolve it?

C++ callback declaration (header):

typedef void (*receiver_t)(char*);
void SetReceiver(receiver_t new_rec);

cpp:

static receiver_t receiver = nullptr;

void SetReceiver(receiver_t new_rec)
{
    printf("setted%i\n", (int)new_rec);
    a = 123;
    if (new_rec != nullptr)
        receiver = new_rec;
}

Cython code:

cdef extern from "TeamSpeak3.h":
    ctypedef void (*receiver_t) (char*) nogil
    cdef void __cdecl SetReceiver(receiver_t new_rec) nogil

cdef void __cdecl NewMessage(char* message) nogil:
    with gil:
        print("hello")

SetReceiver(NewMessage)

Full code: .h http://pastebin.com/ZTCjc6NA

.cpp http://pastebin.com/MeygA8im

.pyx http://pastebin.com/k4X9c54P

.py http://pastebin.com/1YV7tMiF

Artem Selivanov
  • 1,867
  • 1
  • 27
  • 45
  • How do you start the main code that will call the receiver? – michitux Aug 29 '16 at 09:54
  • @michitux I added source links. In `.py` I initialize the cython class from `.pyx`, cython class initializes cpp class from `.h` and `.cpp`, and cpp class calls the TeamSpeak3 API with passing callbacks. Raw simple code. – Artem Selivanov Aug 29 '16 at 10:31

1 Answers1

1

This is a bit of a guess but you probably have a Cython/C/C++ loop running that's holding the GIL and never releasing it. The callback is then forced for wait forever for it.

In normal Python code the GIL is released every few instructions if another thread is waiting for it. In Cython that doesn't happen automatically. One way to ensure that it does happen every so often is to to your loop:

while True:
   # ... do stuff
   with nogil:
      pass

This ensures the GIL is released once per loop.

Unfortunately it's not obvious to me where you have your main loop. I wonder if it's inside connect in your PyTeamSpeak3 class, and perhaps changing the definition of connect to:

def connect(self):
    with nogil:
       self.thisptr.Connect()

might help?

DavidW
  • 29,336
  • 6
  • 55
  • 86
  • Not helps, if used with "nogil" in NewMessage works fine. But I can't do anything python operations... Maybe I can store values to C++ queue with mutex and in main python thread pop all values? But it looks like crutch and smell... :( – Artem Selivanov Aug 29 '16 at 12:05
  • Maybe the answer from http://stackoverflow.com/questions/11687960/how-to-call-a-multi-threaded-c-function-in-cython helps, if the callback is called from another thread I think these additional steps of registering the thread are necessary. Apart from that I can only agree with this answer, I have similar code and it works. – michitux Aug 30 '16 at 12:26
  • @michitux Cython actually generates the code described in [the link](https://docs.python.org/3/c-api/init.html#non-python-created-threads) when you write `with gil:` so it should be OK (I had a similar thought). I still think this answer is the issue, but I don't know where the loop is. – DavidW Aug 30 '16 at 12:41