C code embedded python callback function

Question

C code embedded python callback function，and put data to python queue through callback, when i get data from queue, it's very slow.

Example:

c code like this

static int wrap_func(const int a, const unsigned char *b)
{
  long ret;
  PyObject *arglist;
  PyObject * result = NULL;

  arglist = Py_BuildValue("(s#)", b, a);
  result = PyEval_CallObject(my_callback, arglist);

  /* evaluate result or handle exception */
  ret = PyInt_AsLong(result);
  if (result == NULL)
    return -1;
  Py_DECREF(result);

  return ret;
}

void produce_data()
{
    while(1){
        //produce data to buffer, len is buffer length
        //call callback func 
        wrap_func(buffer, len);
    }
}

compile this c code to so like mywrap.so, and import this so in python python code like this:

import multiprocessing
import mywarp   # mywrap.so

class WorkerThread_a(threading.Thread):
    def __init__(self, workQueue):
        threading.Thread.__init__(self)
        self.workQueue = workQueue
        self.setDaemon(True)
    def run(self):
        while 1:
            try:
                recvdata = self.workQueue.get(block=False)
            except Queue.Empty:
                continue
            #do sth use recvdata

workQueue = multiprocessing.Queue()

def callback_func(a):
    if a:
        workQueue.put(a)
    return 0

def main():
    tmp = WorkerThread_a(workQueue)
    tmp.start()
    mywarp.set_callback(callback_func)
    mywarp.decode_audio()

main()

In python thread, i get data from queue, but i get data very slowly, but in c so, produce data and put to queue through python callback func quickly.

how can i get data from queue quickly like in pure python code.

DavidW · Accepted Answer · 2015-11-27T15:55:13.140

I think what's happening is that your C code is never releasing the global interpreter lock (GIL), so your Python code never has a chance to run. When you're running multiple threads in Python code, they automatically swap the GIL between them and so share time equally, but this doesn't happen without your intervention in C code.

It should probably work a lot better if you acquire and release the GIL once per loop in your C-code (even though you don't do anything that doesn't need it). All I've really done is added the macros Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS at the start of the function.

static int wrap_func(const int a, const unsigned char *b)
{
  Py_BEGIN_ALLOW_THREADS /* ADDED! */
  /* This should give the chance for the main Python thread
  to run, even though no code goes here */
  Py_END_ALLOW_THREADS /* ADDED */

  long ret;
  PyObject *arglist;
  PyObject * result = NULL;

  arglist = Py_BuildValue("(s#)", b, a);
  result = PyEval_CallObject(my_callback, arglist);

  /* evaluate result or handle exception */
  ret = PyInt_AsLong(result);

  /* This is chang */
  if (result == NULL)
    return -1;
  Py_DECREF(result);


  return ret;
}

(I should say - this is an untested guess that I'm 90% sure is right, but I've been wrong before!)

(I've edited this - my original code had the two macros the wrong way round so would not have worked - sorry!) — DavidW, Nov 27 '15 at 15:56
your suggestion is wonderful, it work! Thanks very much. i run code as your suggestion , it work, but when i put the Py_BEGIN_ALLOW_THREADS at the start of the func, and put the Py_END_ALLOW_THREADS before return. i run it , it's segment fault，why is this? — ICYMYM, Nov 28 '15 at 05:04
That's because I made a mistake! (fixed now). You aren't allowed any Python api calls between `Py_BEGIN_ALLOW_THREADS` and `Py_END_ALLOW_THREADS`. So that means any of the functions starting with `Py` won't work. Unfortunately none of this code can actually run in parallel - I've just added a place where it can swap between your C module and your Python code. — DavidW, Nov 28 '15 at 08:31

C code embedded python callback function

1 Answers1