How to use pybind11 in multithreaded application

Question

I want to run Python in a worker thread. However I get strange segfaults and deadlocks of the threads in the worker pool. How do I correctly use pybind11/Python C API to allow the threads to run the jobs?

I know that it does not make much sense to MT python because of the GIL, but thats an intermediate solution to fit the current architecture until there is a better approach.

score 6 · Answer 1 · edited May 14 '20 at 11:33

6

this works. Wrap the long running c++ code with gil_scoped_release and gil_scoped_acquire

pybind11::gil_scoped_release release;

while (true)
{
    // do something and break
}

pybind11::gil_scoped_acquire acquire;

edited May 14 '20 at 11:33

luator

4,769
3
30
51

answered Feb 09 '19 at 19:47

user3713719

325
3
8

Pybind11 has more docs on this [here](https://pybind11.readthedocs.io/en/stable/advanced/misc.html#global-interpreter-lock-gil). – alainh May 06 '21 at 17:04

score 1 · Answer 2 · answered May 24 '18 at 09:17

If you want to do following - From Python you want to run C++ threads to execute different tasks on each thread.

Create a theradpool like https://github.com/progschj/ThreadPool

Write a wrapper class to wrap and bind to Python using PyBind11. Create the instance of ThreadPool and add tasks from Python which in turn creates different threads to execute your tasks.

Disclaimer - I have not tried but this will work :)

How to use pybind11 in multithreaded application

2 Answers2