Python GIL and threads

Question

I have embedded Python3 in my big C++ application. Python gives the user script capability for custom data processing.
Problem : I have many threads that interact with Python and I don't really get how to protect my code with GIL. So far, the only way I made my code work is using boost::mutex.

Here is a very simplified example that reproduces my problem:

Thread A calls first Init() to initialize Python (static function).
Thread B calls Pythonize() to do some work on Python. Thread B is blocked on the first call for locking GIL.

Code:

#include <iostream>
#include <boost/thread.hpp>
#include <boost/bind.hpp>

#include "Python.h"

struct RTMaps_GILLock
{
    RTMaps_GILLock()
    {
        std::cout << "Locking..." << std::endl;
        m_state = PyGILState_Ensure();
    }

    ~RTMaps_GILLock()
    {
        std::cout << "Unlocking..." << std::endl;
        PyGILState_Release(m_state);
    }

private:
    PyGILState_STATE m_state;
};
#define GILLOCK RTMaps_GILLock lock;

class PythonEmbed
{
public:
    static void Init()
    {
        Py_Initialize();
        // EDIT : adding those two lines made my day :
        PyEval_InitThreads(); // This acquires GIL
        PyEval_SaveThread(); // Release the GIL
    }

    void Pythonize()
    {
        GILLOCK;
        // Never goes here :(
        std::cout << "OK" << std::endl;
    }
};

int main()
{
    PythonEmbed::Init();

    PythonEmbed pyt;
    boost::thread t(boost::bind(&PythonEmbed::Pythonize, pyt));

    t.join();
}

it is deadlocking in the first lock call. The console shows: Locking...

The "OK" is never printed. What am I doing wrong ?

EDIT : corrected code, now it is working. I needed to release the GIL from the main thread.

Are these the only threads involved? Generally, when Python calls into C, the GIL is already acquired. Is it possible you have another thread somewhere which blocks inside of C, without explicitly acquiring the GIL? — Kevin, Mar 17 '15 at 14:57
@Kevin Yes. BTW, Python has no chance to call into C. Until I deadlock, I have only time to init python in thread A, and when thread B arrives to do some work it is stuck in the first lock. Actually, thread A won't touch Python until the end of my application anymore, is there anything to do ? I've read the documentation 1000 times and what I am doing should work... Already hating GIL. — poukill, Mar 18 '15 at 10:28
@Kevin I rewrote my post to show a full example that everyone can compile and test. This example is failing. — poukill, Mar 18 '15 at 17:20

score 4 · Accepted Answer · answered Mar 18 '15 at 11:16

I had your exact problem, be sure to not call PyGILState_Ensure() from the main thread, the one that initialize Pythons, because it needs a total different call. I've end setting a thread mapper, and each call to my acquirePython() checks what thread is calling it, if it's the main thread , it uses:

PyEval_SaveThread();

otherwise it stores the GIL. Those are the relevant sections of my class:

void MManager::acquirePython(void) {
    MThread thisThread = MFramework::MProcesses::GetCurrentThread();
    if (thisThread != mainThread) {
        Lock();
        std::map<MThread,void*>::iterator i = threadStates.find(thisThread);
        if (i == threadStates.end()) {
            Unlock();
            PyGILState_STATE gstate = PyGILState_Ensure();
            _PyGILState_STATE_* encState = new _PyGILState_STATE_;
            encState->state = gstate;
            encState->refCount = 1;
            Lock();
            threadStates[thisThread] = encState;
            Unlock();
        } else {
            _PyGILState_STATE_* encState = (_PyGILState_STATE_*)i->second;
            encState->refCount = encState->refCount + 1;
            Unlock();
        }

    } else {
        if (mainThreadState) PyEval_RestoreThread((PyThreadState*)mainThreadState);
    }

}

void MManager::releasePython(void) {
    MThread thisThread = MFramework::MProcesses::GetCurrentThread();
    if (thisThread != mainThread) {
        Lock();
        std::map<MThread,void*>::iterator i = threadStates.find(thisThread);
        if (i != threadStates.end()) {
            _PyGILState_STATE_* encState = (_PyGILState_STATE_*)i->second;
            if (encState->refCount <= 1) {
                threadStates.erase(i);
                Unlock();

                PyGILState_Release(encState->state);
                delete encState;
            } else {
                encState->refCount = encState->refCount - 1;
                Unlock();
            }
        } else {
            Unlock();
        }

    } else {
        mainThreadState = PyEval_SaveThread();
    }
}

I've completely updated my post to a simpler example that you can compile.The GIL lock is done on a different thread than the main thread. So I should not fall into the issue you described — poukill, Mar 18 '15 at 17:21
you need to call PyEval_InitThreads(); in your main function, from the main thread that calls PyInitialize() — Leonardo Bernardini, Mar 18 '15 at 17:31
@LeonardoBernardini: And *then*, OP needs to release the GIL from the main thread, since InitThreads acquires it. — Kevin, Mar 18 '15 at 18:08
You were right, I needed to call `PyEval_SaveThread` to release the GIL from the main thread. — poukill, Mar 19 '15 at 09:58

Python GIL and threads

1 Answers1