How to iterate through all python objects from a c extension?

Question

I'm working on a python memory profiler where I collect the size of python objects with the following method:

sum(map(sys.getsizeof, gc.get_objects()))

This is significantly the slowest part of the code - especially gc.get_objects - so I decided to speed it up and rewrite it as a c extension. The problem is that the python c API doesn't give access to the gc modules internal data what is used by gc.get_objects.

Is it possible to iterate through all objects using the c API, without calling the expensive gc.get_objects?

I think you will need to patch the `gcmodule.c` for that. A version of `append_objects` that simply adds up the sizes of the objects would be ideal. — Dan D., Sep 24 '15 at 08:31
I'm going to note, the cost of calling `gc.get_objects()`, beyond the direct manipulation of internals, is just a matter of constructing a `list` with all the objects, which, while more expensive (and requiring a wait before processing) than direct iteration, is a fairly small cost (if you've got 100,000 `gc` tracked objects, the total cost of collecting the objects should be ~1 millisecond, with the list overhead itself being well under half the total cost). Have you tried just calling `gc.get_objects()` and having the C extension do the work with the resulting list? — ShadowRanger, Sep 24 '15 at 15:07
Additional note: The gc tracked objects don't include all objects, only objects that can be responsible for a reference cycle (which excludes immutable objects that can't store references to non-immutable objects, e.g. `int`, `float`, `str`, etc.), so if your goal is to track all object allocation/deallocation, the `gc` module won't help. If you're on Py3.4 or higher, take a look at the [`tracemalloc` module](https://docs.python.org/3/library/tracemalloc.html), which is more helpful/comprehensive/fine-grained than just "tell me all the GC tracked objects now". — ShadowRanger, Sep 24 '15 at 15:20
Lastly, if this is on Python 2 rather than 3, you could probably get a decent speed up by replacing `map` with `itertools.imap` (or adding `from future_builtins import map` to the top of your code) in your summation function, since it will avoid creating a complete list of sizes before summing, and instead just compute and sum as it goes. On Python 3, `map` is already a generator function, so no changes are needed. — ShadowRanger, Sep 24 '15 at 15:23
@DanD., @ShadowRanger, thanks the useful thoughts. Seems, the fastest solution - which doesn't require python patching - is calling `gc_get_objects()` directly from a C extension. — asciimoo, Sep 28 '15 at 07:37

score 0 · Answer 1 · answered Nov 05 '22 at 04:48

Here is code I found on GitHub

https://github.com/crazyguitar/pysheeet/blob/master/docs/notes/python-c-extensions.rst#iterate-a-list

#include <Python.h>

#define PY_PRINTF(o) \
    PyObject_Print(o, stdout, 0); printf("\n");

static PyObject *
iter_list(PyObject *self, PyObject *args)
{
    PyObject *list = NULL, *item = NULL, *iter = NULL;
    PyObject *result = NULL;

    if (!PyArg_ParseTuple(args, "O", &list))
        goto error;

    if (!PyList_Check(list))
        goto error;

    // Get iterator
    iter = PyObject_GetIter(list);
    if (!iter)
        goto error;

    // for i in arr: print(i)
    while ((item = PyIter_Next(iter)) != NULL) {
        PY_PRINTF(item);
        Py_XDECREF(item);
    }

    Py_XINCREF(Py_None);
    result = Py_None;
error:
    Py_XDECREF(iter);
    return result;
}

static PyMethodDef methods[] = {
    {"iter_list", (PyCFunction)iter_list, METH_VARARGS, NULL},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef module = {
    PyModuleDef_HEAD_INIT, "foo", NULL, -1, methods
};

PyMODINIT_FUNC PyInit_foo(void)
{
    return PyModule_Create(&module);
}

How to iterate through all python objects from a c extension?

1 Answers1