Why do Python functions get garbage collected?

Question

I have a C++ library which uses Python callbacks. The callback, i.e. PyObject*, is stored in an object of class UnaryFunction, and the constructor Py_INCREFs it. The destuctor Py_XDECREFs it. That's the problem. The interpreter segfaults on that DECREF.

My solution is to just not DECREF it, but that seems wrong. What is the proper way to INC/DEC the reference count of a function, and more importantly, why does the interpreter try to GC a function body when there are other live references to it?

Edit: On Linux instead of a segfault I get an assertion fail that says:

python: Objects/funcobject.c:442: func_dealloc: Assertion 'g->gc.gc_refs != (-2)' failed.

This seems like a question for the mailing list. Google "Python mailing list". On there you can find people who know Python very well, sometimes even the creators/devs of Python themselves. — Sunjay Varma, Sep 08 '11 at 02:06
related: http://stackoverflow.com/questions/7326762/cython-callback-works-correctly-for-function-but-not-for-bound-method/7339258#7339258 — jfs, Sep 08 '11 at 03:21
@J.F.Sebastian, good guess but the method does not go out of scope. The problem is exhibited in a simple flat script with a simple `def myfunc(x):". If I comment out the parts that use the callback then I can call myfunc() through the end of the script. — Adam, Sep 08 '11 at 20:45

score 1 · Answer 1 · edited Sep 08 '11 at 11:13

1

A crash does not necessary mean that it is trying to GC an used object. It can also mean that you are calling python code without the interpretor lock.

Calling Py_XDECREF in a destructor leads me to think you have something like this:

void MyCallback(myfunc, myarg)
{
    ...
    PyGILState_STATE gilstate = PyGILState_Ensure();
    try {
            myfunc(myarg);
    } catch (...) {
        ...
    }
    PyGILState_Release(gilstate);

    // myfunc goes out of scope here --> CRASH because we no longer own the GIL
}

with the simple solution:

...
try {
    scopefunc = myfunc;
    myfunc = emptyfunc();
    scopefunc(myarg);
} ...

edited Sep 08 '11 at 11:13

brandizzi

26,083
8
103
158

answered Sep 08 '11 at 03:47

Mihai Stan

1,052
6
7

Actually, I do not touch the GIL at all. From what I understand the GIL does not get released unless it's explicitly released. My C++ code simply uses PyEval_CallObject() to call the function. I say it's collected because the segfault is: `Objects/funcobject.c:442: func_dealloc: Assertion 'g->gc.gc_refs != (-2)' failed.` and that's talking about a dealloc and GC references. – Adam Sep 08 '11 at 20:36
If you have just one thread then things should be simpler. You have to find the mismatched incref/decref, and calling `sys.getrefcount` before and after each step that involves the function could give you some clues – Mihai Stan Sep 08 '11 at 21:24

score 0 · Accepted Answer · answered Sep 09 '11 at 00:12

0

It appears the Py_INCREF simply doesn't actually increment the refcount.

answered Sep 09 '11 at 00:12

Adam

16,808
7
52
98

Why do Python functions get garbage collected?

2 Answers2