NumPy C API extension causes excessive memory usage

Question

I have written a C extension to NumPy to speed up some of my calculations, but I am getting increasing memory usage as I call the function repeatedly. I have trimmed down the function into a minimum example:

PyObject* memory_test_function(PyObject* self, PyObject* args)
{
    PyArrayObject *ang;
    int i;

    if (!PyArg_ParseTuple(args, "O", &ang)) return NULL;

    int L0 = (int) PyArray_DIMS(ang)[0];

    // ballooning memory usage 
    npy_intp final_out_dims[2] = {L0,1};
    PyObject *output_array;
    output_array = PyArray_SimpleNew(2, final_out_dims, NPY_FLOAT64);
    Py_INCREF(output_array);

    for (i=0;i<L0;i++) 
    {
        *(double *)PyArray_GETPTR2(output_array,i,0) = 
            tan(*(double *)PyArray_GETPTR2(ang,i,0));
    }

    return PyArray_Return(output_array);

    /* constant memory usage
    double sum=0.0;
    for (i=0;i<L0;i++) sum+=tan(*(double *)PyArray_GETPTR2(ang,i,0));
    return PyFloat_FromDouble(sum); */
}

The problem appears to be caused by creating the output array, as just returning a float without creating the array object is constant in memory. I suspect there is a problem with INCREF/DECREF, but I thought I was doing everything correctly. Calling this function repeatedly (a million or so times) makes memory usage increase linearly with time, which makes me think something is not being garbage collected correctly. Using gc manually doesn't help. Please let me know if there is anything obvious I'm missing!

I know nothing about the Python C API but I’m suspicious about your `Py_INCREF` call: You’re not sharing the reference with any other object here, you’re *passing* it to the caller. So either don’t increase the reference count at all, or decrease it once you’re done working with it locally. You can check this in your calling code via `gc.get_referrers()`. If I’m right, this will return 2 if you haven’t made any other copies, and it *should* return 1 (you can also check it in your C code via `Py_REFCNT(output_array)`). — Konrad Rudolph, Jul 02 '19 at 12:53
Indeed that fixes the problem! I didn't realize passing it doesn't require INCREF. I'd be happy to accept this as the answer. — Anger Density, Jul 02 '19 at 12:58

score 1 · Accepted Answer · answered Jul 02 '19 at 12:59

PyArray_SimpleNew calls (indirectly, probably) _Py_NewReference under the hood. This function sets the reference count on the newly created reference to 1.

Your subsequent Py_INCREF increases the ref count to 2 and thus makes sure that Python will never free this object, even if all references to it cease to exist, because its reference count will never drop to 0.

The Py_INCREF call is unnecessary because you’re not sharing the reference with any other object here, you’re just using it locally and then passing it to the caller.

NumPy C API extension causes excessive memory usage

1 Answers1