How to use Py_AddPendingCall

Question

I have an embedded Python program which runs in a thread in C.

When the Python interpreter switches thread context (yielding control to another thread), I'd like to be notified so I can perform certain necessary operations.

It seems that Py_AddPendingCall is exactly what I'm looking for. However, the API docs are pretty brief on this function, and I'm confused as to how Py_AddPendingCall is supposed to be used. From reading the docs, my understanding is that:

A worker thread calls Py_AddPendingCall and assigns a handler function.
When the Python interpreter runs, it calls the handler function whenever it yields control to another thread
The handler itself is executed in the main interpreter thread, with the GIL acquired

I've googled around for example code showing how to use Py_AddPendingCall, but I can't find anything. My own attempt to use it simply doesn't work. The handler is just never called.

My worker thread code:

#include <pthread.h>
#include <stdlib.h>
#include <stdio.h>

const char* PYTHON_CODE = 
"while True:\n"
"   for i in range(0,10): print(i)\n"
"\n";

int handler(void* arg)
{
    printf("Pending Call invoked!\n");
    abort();
}

void* worker_thread(void*)
{
    PyGILState_STATE state = PyGILState_Ensure();
    int res = Py_AddPendingCall(&func, nullptr);
    cout << "Result: " << res << endl;
    PyRun_SimpleString(CODE);
    PyGILState_Release(state);

    return 0;
}

int main()
{
    Py_Initialize();
    PyEval_InitThreads();
    PyEval_ReleaseLock();

    pthread_t threads[4];
    for (int i = 0; i < 4; ++i)
        pthread_create(&threads[i], 0, worker_thread, 0);

    for (int i = 0; i < 4; ++i) pthread_join(threads[i], 0);

    Py_Finalize();
}

In this example, worker_thread is invoked in C as a pthread worker thread. In this test I run 4 worker threads, so some context switching should happen. This loops infinitely, but the pending call handler is never invoked.

So, can someone provide a minimal working example that shows how Py_AddPendingCall is supposed to be used?

@nneoneo - it is called from a `pthread` worker thread via `pthread_create` — Channel72, Sep 16 '12 at 16:55
Does your main thread execute Python code at all? Are you embedding or extending Python? — nneonneo, Sep 16 '12 at 17:00
I'm embedding Python - the worker threads call Python code, but the main thread doesn't. — Channel72, Sep 16 '12 at 17:03
Please post a _complete_ example that fails to work. I tried your code, modified to include a `main()` that initializes Python and call test_thread(), and it invoked the handler. I then modified it to actually start a thread, and invoke test_thread in the other thread, and it still invoked the handler. See http://pastebin.com/pQrnHEFX — user4815162342, Sep 16 '12 at 17:07
@user4815162342: Your example doesn't initialize any Python threads -- there's still only one thread in Python's view. — nneonneo, Sep 16 '12 at 17:12
A complete example would still be useful. Your snippet is misleading in that it looks like it's (almost) complete, while it requires further setup to test. — user4815162342, Sep 16 '12 at 17:18
What sort of necessary operations need to be performed on every context switch between threads? Seems like an unnecessary overhead... — nneonneo, Sep 16 '12 at 17:38

nneonneo · Accepted Answer · 2012-09-16T17:37:48.763

Pending calls are only executed on the main thread, and the main thread can only service pending calls when it executes Python code. So, the main thread must be running an interpreter loop in order to service any pending calls. From Python/ceval.c:

Py_MakePendingCalls(void)
{
    ...

    /* only service pending calls on main thread */
    if (main_thread && PyThread_get_thread_ident() != main_thread)
        return 0;

    ...
}

In an embedding scenario, the "main thread" is whichever thread calls PyEval_InitThreads (note that it is also automatically called the first time you start a thread from Python code). Since your main thread sits in pthread_join, it is not executing Python code and therefore not dispatching pending calls.

Note too that the handler only executes when control is handed to the main thread -- the handler doesn't run if control goes from one worker to another. So, Py_MakePendingCalls is probably not an appropriate interface to use for this task.

How to use Py_AddPendingCall

1 Answers1