1

I have registered a python callback with a dll using the ctypes library. When the callback is triggered, i try to free up an asyncio future i have set up. Since the callback happens in a separate thread that gets spawned by the dll, i use the loop.call_soon_threadsafe() function to get back to the eventloop that started it all.

Mostly this works fine, but every once in a while the future fails to be unblocked. In the minimal example here this also happens sometimes, but here i see that in those cases the callback doesn't even arrive (or at least the corresponding print doesn't happen). I tried this only with python 3.8.5 so far. Is there some race condition here that i did not notice?

Here's a minimal example:

import asyncio
import os

class testClass:
    loop = None
    future = None
    exampleDll = None

    def finish(self):
        #now in the right c thread and eventloop.
        print("callback in eventloop")
        self.future.set_result(999)

    def trampoline(self):
        #still in the other c thread
        self.loop.call_soon_threadsafe(self.finish)

    def example_callback(self):
        #in another c thread, so we need to do threadsafety stuff
        print("callback has arrived")
        self.trampoline()
        return

    async def register_and_wait(self):
        self.loop = asyncio.get_event_loop()
        self.future=self.loop.create_future()
        callback_type = ctypes.CFUNCTYPE(None)
        callback_as_cfunc = callback_type(self.example_callback)
        #now register the callback and wait
        self.exampleDll.fnminimalExample(callback_as_cfunc, ctypes.c_int(1))
        await self.future
        print("future has finished")

    def main(self):
        path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "minimalExample.dll")
        #print(path)
        ctypes.cdll.LoadLibrary(path)
        #for easy access
        self.exampleDll = ctypes.cdll.minimalExample
        asyncio.run(self.register_and_wait())

if __name__ == "__main__":
    for i in range(0,100000):
        print(i)
        test = testClass()
        test.main()

You can get the compiled example dll and its source from the repository here to reproduce.

skoster
  • 11
  • 2
  • `immediateCallback` and `eventuallyCallback` are called from another thread, but they don't acquire the GIL before calling into Python and release it afterward. See [the documentation](https://docs.python.org/3/c-api/init.html#non-python-created-threads) for an explanation. – user4815162342 Apr 14 '21 at 07:14
  • I assumed ctypes did this automatically before it goes into the actual python code, as implied in its [documentation](https://docs.python.org/3/library/ctypes.html#callback-functions) where it says "...if the callback function is called in a thread created outside of Python’s control (e.g. by the foreign code that calls the callback), ctypes creates a new dummy Python thread on every invocation." And the answer to [this question](https://stackoverflow.com/questions/34686826/calling-python-function-from-c-as-a-callback-what-is-the-right-way-to-handle-th) says so explicitly but it may be wrong. – skoster Apr 14 '21 at 14:40
  • OK then, I missed the part that the Python function gets invoked through a ctypes trampoline that handles this for you. But then I fail to see what other issue your code might be having - good luck! – user4815162342 Apr 14 '21 at 15:15

1 Answers1

0

The issue (at least in this minimal example) does not show up any more if i reuse the same eventloop instead of spawning a new one for every iteration with asyncio.run

The problem is thus fixed, but it doesn't feel right.

skoster
  • 11
  • 2