Concurrency: Are Python extensions written in C/C++ affected by the Global Interpreter Lock?

Question

One of Python's strongest points is the ease of writing C and C++ extensions to speed up processor intensive parts of the code. Can these extensions avoid the Global Interpreter Lock or are they also restricted by the GIL? If not, then this "ease of extension" is even more of a killer feature than I previously realized. I suspect the answer is not a simple yes-or-no but I am not sure, so I am asking the question here on StackOverflow.

Answer 1

Yes, calls to C extensions (C routines called from Python) are still subject to the GIL.

However, you can manually release the GIL inside your C extension, so long as you are careful to re-assert it before returning control to the Python VM.

For information, take a look at the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros: http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock

Answer 2

C/C++ extensions to Python are not bound by the GIL. However, you really need to know what you're doing. From http://docs.python.org/c-api/init.html:

The global interpreter lock is used to protect the pointer to the current thread state. When releasing the lock and saving the thread state, the current thread state pointer must be retrieved before the lock is released (since another thread could immediately acquire the lock and store its own thread state in the global variable). Conversely, when acquiring the lock and restoring the thread state, the lock must be acquired before storing the thread state pointer.

Why am I going on with so much detail about this? Because when threads are created from C, they don’t have the global interpreter lock, nor is there a thread state data structure for them. Such threads must bootstrap themselves into existence, by first creating a thread state data structure, then acquiring the lock, and finally storing their thread state pointer, before they can start using the Python/C API. When they are done, they should reset the thread state pointer, release the lock, and finally free their thread state data structure.

Answer 3

If you’re writing your extension in C++, you can use RAII to easily and legibly write code manipulating the GIL. I use this pair of RAII structlets:

namespace py {

    namespace gil {

        struct release {
            PyThreadState* state;
            bool active;

            release()
                :state(PyEval_SaveThread()), active(true)
                {}

            ~release() { if (active) { restore(); } }

            void restore() {
                PyEval_RestoreThread(state);
                active = false;
            }
        };

        struct ensure {
            PyGILState_STATE* state;
            bool active;

            ensure()
                :state(PyGILState_Ensure()), active(true)
                {}

            ~ensure() { if (active) { restore(); } }

            void restore() {
                PyGILState_Release(state);
                active = false;
            }
        };

    }

}

… allowing the GIL to be toggled for a given block (in a semantic manner that may seem dimly familiar for any context-manager Pythonista fans):

PyObject* YourPythonExtensionFunction(PyObject* self, PyObject* args) {

    Py_SomeCAPICall(…);     /// generally, if it starts with Py* it needs the GIL
    Py_SomeOtherCall(…);    /// ... there are exceptions, see the docs

    {
        py::gil::release nogil;
        std::cout << "Faster and less block-y I/O" << std::endl
                  << "can run inside this block -" << std::endl
                  << "unimpeded by the GIL";
    }

    Py_EvenMoreAPICallsForWhichTheGILMustBeInPlace(…);

}

… Indeed, personally also I find the ease of extending Python, and the level of control one has over the internal structures and state, a killer feature.

Answer 4

Check out Cython, it has similar syntax to Python but with a few constructs like "cdef", fast numpy access functions, and a "with nogil" statement (which does what it says).