Memory Leaks and Reference Counting in Python/C API

Question

I am pretty new to Python and its C API. I still do not understand how reference counting works. I have written a module for particle tracking that exposes to python a number of C++ thread tracking functions that I had written and tested in the past. (I far as I can tell they do not have memory leaks themselves).

When I call one of these function from within Python repeatedly I can see that the memory usage is growing slowly. I believe there is a memory leak somewhere (probably everywhere :O) I copied below the relevant fragment of the main tracking function so that someone could point to me whether I should make calls to Py_DECREFs (on item_py, for example?)

PyObject* _track_particles() {

        // more code here ... (no Python/C API  calls) 

        PyObject* result_py = PyTuple_New(particles.size());
        for(int i=0; i<particles.size(); ++i) {
            PyObject* item_py = PyTuple_New(2);
            if (lost_at_turn_idx[i] == PARTICLE_NOT_LOST) {
                int offset = i * (nr_turns+1) * 6 + nr_turns * 6;
                PyTuple_SetItem(item_py, 0, Py_True);
                PyTuple_SetItem(item_py, 1, Py_BuildValue("(dddddd)", 
                            data_out[offset + rx], data_out[offset + px],   
                            data_out[offset + ry], data_out[offset + py],
                            data_out[offset + de], data_out[offset + dl]));
            } else {
                PyTuple_SetItem(item_py, 0, Py_False);
                PyTuple_SetItem(item_py, 1, 
                        Py_BuildValue("(ii)", lost_at_turn_idx[i], 
                                              lost_at_element_idx[i]));
            }
            PyTuple_SetItem(result_py, i, item_py);
        }       
        return result_py;
    }

ps: found this reference usefull

Answer 1

Is this relevant?

Use t = PyTuple_New(n) instead, and fill it with objects using PyTuple_SetItem(t, i, o) – note that this "eats" a reference count of o, so you have to Py_INCREF() it.

I'm not entirely sure whether this passage is clear, but it may be a good starting point.

Answer 2

Edit: Many thanks to @DavidW for pointing this out to me (see comments). You definitely need to Py_INCREF Py_True and Py_False. C does weird stuff when you do stuff you're not supposed to do, but it's hard to say if this might be causing your memory issues.

Several years late, but honestly, your code seems fine . result_py is being returned and thus only needs to be Py_DECREF'd by the caller of _track_particles (I assume there is a caller for _track_particles within your C++ code since there is PyObject *self, PyObject *args defined in its signature) while item_py references are being stolen with PyTuple_SetItem. The rest of your new references are being stolen by PyTuple_SetItem so that's not a problem either. Interestingly, you have given a code fragment where not a single Py_DECREF is needed.

How are you checking for your memory leak? I know this is old, but Python 3 comes with tracemalloc so you could wrap a function call with that to check.