Call python in parallel from C

Question

I need to call a Python function from my C code. It works perfectly, but when I want to do parallelization, it breaks down. Please see the following minimal C code:

#include <Python.h>
#include <stdio.h>

int main(void)
{
  double Z = 1.;
  double k = 1.;
  double l = 1.;
  double eta = -Z/k;

  Py_Initialize();

  PyObject* pName = PyString_FromString("mpmath");
  PyObject* pModule = PyImport_Import(pName);
  PyObject* pFunc = PyObject_GetAttrString(pModule, "coulombf");

  PyObject* pl = PyFloat_FromDouble(l);
  PyObject* peta = PyFloat_FromDouble(eta);

  int i;
#pragma omp parallel for private(i)
  for(i=0; i<10000; i++)
  {
    double r = 0.01*i;
    PyObject* prho = PyFloat_FromDouble(k*r);
    PyObject* pArgs = PyTuple_Pack(3, pl, peta, prho);
    PyObject* pValue = PyObject_CallObject(pFunc, pArgs);
    double value = PyFloat_AsDouble(pValue);
    printf("r=%.2f\tf=%.6f\n",r,value);
  }

  Py_Finalize();
  return 0;
}

Let's name this file as testPython.c, you can compile it with gcc -fopenmp testPython.c -o testPython -I/usr/include/python2.7 -L/usr/lib64/python2.7/config -lpython2.7.

Now run it with ./testPython, you with see such error: Fatal Python error: GC object already tracked. (Sometimes, the error info differs.)

But if you compile it leaving -fopenmp out, the program works perfectly.

How can I overcome this problem? Thanks!

Edit:

As answered by Natecat, John Bollinger, and Olaf, multithreading is unlikely to speed up the process much, but multiprocessing can really speed up the computation. The pure python script is as simple as following:

import numpy
from mpmath import coulombf
from multiprocessing import Pool

Z = 1.
k = 1.
l = 1.
eta = -Z/k

def coulombF(r):
    return coulombf(l,eta,k*r)

pool = Pool(12)
result = pool.map_async(coulombF, numpy.arange(0.,100.,0.01))
print(result.get())

But how do I do it in C? I haven't found the way yet.

Answer 1

@Natecat's answer is basically right, if a bit lacking in detail and nuance. The docs of Python's C API give a more complete picture. Supposing that this is the Python implementation you are using, you need to be aware of the following:

The Python interpreter is not fully thread-safe. In order to support multi-threaded Python programs, there’s a global lock, called the global interpreter lock or GIL, that must be held by the current thread before it can safely access Python objects. Without the lock, even the simplest operations could cause problems in a multi-threaded program [...].

Therefore, the rule exists that only the thread that has acquired the GIL may operate on Python objects or call Python/C API functions. In order to emulate concurrency of execution, the interpreter regularly tries to switch threads (see sys.setswitchinterval()). The lock is also released around potentially blocking I/O operations like reading or writing a file, so that other Python threads can run in the meantime.

and

when threads are created from C (for example by a third-party library with its own thread management), they don’t hold the GIL, nor is there a thread state structure for them.

Note: this is exactly the case with OpenMP.

If you need to call Python code from these threads [...] you must first register these threads with the interpreter by creating a thread state data structure, then acquiring the GIL, and finally storing their thread state pointer, before you can start using the Python/C API. When you are done, you should reset the thread state pointer, release the GIL, and finally free the thread state data structure.

The PyGILState_Ensure() and PyGILState_Release() functions do all of the above automatically. The typical idiom for calling into Python from a C thread is:

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);

You must implement that pattern to allow multiple OpenMP threads safely to make concurrent calls into the same CPython interpreter, but you are unlikely to get much benefit from the parallelization, as the various OpenMP threads will largely be prevented from running concurrently.

Answer 2

Any type of true multithreading (E.G. using multiple system threads in one process) in python is not possible, at least in the most common python implementations. You can either not use any type of paralellization or you can switch to an implementation without a GIL. Here is an article with more information on the subject: https://wiki.python.org/moin/GlobalInterpreterLock