1

I am wrapping library that uses global defined function as callback.

void library_callback(size_t id, char *buf, size_t len);

This library_callback function is called internally by library but I can set this function via public API.

I want to create list of listeners (Python callable objects) that are called on every callback execution.

static PyObject *listeners = NULL; // initialize later as PyList_New(...)

void library_callback(size_t id, char *buf, size_t len) {
    for (size_t i = 0, count = PyList_Size(listeners); i < count; i++) {
        PyObject *listener = PyList_GetItem(listeners, i);
        // call listener with id and buf
    }
}

It works very well till I wrap library_callback with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS. But without Py_*_ALLOW_THREADS library is working in blocking mode (even when I try to execute Thread(target=myfunc)).

I created simple method to check whether problem is with static variable or not.

PyObject *TestObject_Test(PyObject *self, PyObject *noargs) {
    static PyObject *variable = NULL;
    if (variable == NULL) {
        variable = PyUnicode_FromString("variable");
    }
    Py_BEGIN_ALLOW_THREADS
    printf("variable\t%s\n", PyUnicode_AsUTF8(PyUnicode_FromFormat("%R", variable)));
    Py_END_ALLOW_THREADS
    return Py_INCREF(variable), variable;
}

Then after code below I got fish: 'python3' terminated by signal SIGSEGV (Address boundary error)

obj = TestObject()
obj.test()

How can I write this code properly and get feature to run callbacks in background?

ventaquil
  • 2,780
  • 3
  • 23
  • 48
  • 2
    "It works very well till I wrap library_callback with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS" - well, yeah. Why are you doing that? You can't do that. You're releasing the GIL while you still need it. – user2357112 Nov 05 '20 at 21:33
  • @user2357112supportsMonica I don't understand. Without `Py_*_ALLOW_THREADS` I cannot run my function in the background (using `Thread(target=func)` for example). How can I achieve "running-in-background" without `Py_*_ALLOW_THREADS`? I am new in Python C environment so any tips would be nice! – ventaquil Nov 05 '20 at 21:36
  • Whatever is causing your code to block, we can't tell from this, but the solution is not to release a lock while you actively need it. – user2357112 Nov 05 '20 at 21:40
  • If your listeners are written in Python, they will automatically, periodically release the GIL at safe points to give other threads a chance to run. – user2357112 Nov 05 '20 at 21:42
  • Listeners are lambdas or something that is callable (called via `PyObject_Call` function). So I should ignore fact that I can wait a while for result? Because I cannot do anything with that? – ventaquil Nov 05 '20 at 21:45
  • 1
    `PyUnicode_AsUTF8` and `PyUnicode_FromFormat` definitely can't appear in a `Py_BEGIN_ALLOW_THREADS` block because they're interacting with Python objects. – DavidW Nov 06 '20 at 10:05
  • @DavidW nice to know. I couldn't find it in docs. – ventaquil Nov 10 '20 at 07:18

0 Answers0