0

I am writing a modular Python program for data acquisition. I compiled different codes for cameras (the "producers") and for the data analyzing (the "consumers"), and made Python modules from them using PyBind11. Each module has a C++ method Start(), exposed to Python, which starts independent operations in separate threads.

My Python script loads one of the "producer/camera" module and one of the "consumer/analyzer" module. It allocates memory by creating a numpy array and passes the pointers to that memory both to the producer and the consumer, using PyBind11 to expose the underlying pointers. Now the producer writes on that buffer and the consumer reads from that buffer. That works well.

Now, to avoid busy waiting from the consumer, I would also like to share mutex and condition_variable betweens the modules. However, PyBind11 does not seem to have casting predefined for std::mutex and std::condition_variable.

How can I simply create a mutex and a condition_variable in Python and pass it to python modules written in C++ ? Or, how can I create a mutex and a condition_variable in the producer module and pass to the consumer module a pointer towards them through Python ?

I've tried the following on the PyBind11 bindings.cpp side and the code compiles :

MyClass::define_cv_mutex(std::condition_variable *cv, std::mutex *mtx)
{
cv_pointer = cv; //cv_pointer is member of MyClass
mtx_pointer = mtx; //mtx_pointer is member of MyClass
}

PYBIND11_MODULE(bindings, m){

pybind11::class_<MyClass>(m, "MyClass")`
.def(pybind11::init<>())`
.def("define_cv_mutex",&MyClass::define_cv_mutex)`
}

On the Python side I naively tried :

mutex = threading.Lock() 
cv = threading.Condition() 
A = MyClass()  
A.define_cv_mutex(cv,mutex)

but it does not work

TypeError: define_cv_mutex(): incompatible function arguments. The following argument types are supported: 1. (self: MyModule.bindings.MyClass,arg0: std::condition_variable, arg1: std::mutex) -> None Invoked with: <MyModule.bindings.MyClass object at 0x7f0d7700bdf0>,<Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f0d5f995200>, 0)>, <unlocked _thread.lock object at 0x7f0d5928a940>

magtweezers
  • 31
  • 5
  • 2
    Can't use just create a wrapper with the same functionality that delegates the calls to mutex/cv? – ALX23z May 14 '23 at 07:27
  • @ALX23z so I tried the following : both my modules expose the class MutexCV which contains a mutex and a CV. MyClassA from ModuleA has a an exposed method get_MutexCV() and MyClassB from ModuleB has an exposed method set_MutexCV. But when I do in python A.set_MutexCV(B.get_MutexCV) Python does not recognize that both MutexCV are of the same type because they are defined in two different modules. I get an TypeError, MyModuleB.MutexCV is incompatible with function taking as argument MyModuleA.MutexCV. I don't know how force the cast. – magtweezers May 14 '23 at 17:02
  • 1
    you ought to define MutexCV in one place. And have both modules use the same one. – ALX23z May 14 '23 at 18:11
  • 2
    Alternatively, you can create an interface mutex/cv class that can work with both implementations. – ALX23z May 14 '23 at 18:14
  • I want the two different modules to build independently of each other and to be importable independently, so I understand that I must define my `MutexCV` class in both of them. I tried to not expose the `MutexCV` class with PyBind11 in the _taker_ module (the one with `set_MutexCV(MutexCV *mcv)`), and only expose the method `set_MutexCV`. this works in Ubuntu but ... the TypeError remains in Windows strangely. How can I create an interface class ? Can I do that in Python ? I would like the interfacing to be pure Python. – magtweezers May 14 '23 at 20:01
  • You can always put the MutexCV into another module and have both modules import it. Even with the interface you need something like this. – ALX23z May 14 '23 at 20:14
  • 1
    See this link on python interfaces. https://realpython.com/python-interface/ – ALX23z May 14 '23 at 20:17
  • Ok it is a bit complicated for my application, isn't there a simpler way to share opaque pointers between C++ modules through Python ? I wrote a more general question, because I understand now that my question is more general than a `mutex` or a `condition_variable` question, hopefully it is more clear : https://stackoverflow.com/questions/76272413/pybind11-share-opaque-pointers-between-different-c-modules – magtweezers May 17 '23 at 13:17

0 Answers0