0

I'm trying to deal with an API that returns an array of PyObjects and I want to extend an existing python list with it's contents.

So far I'm just appending every item of the array:

static PyObject *
myfunc(PyObject *module, PyObject *alist, PyObject **items, Py_ssize_t arrsize)
{
    Py_ssize_t i;
    for (i = 0 ; i < arrsize; i++) {
        if (PyList_Append(alist, items[i]) == -1) {
            return NULL;
        }
    }
    return alist;
}

But a list is more or less just a wrapper around an array of PyObjects itself.

So is there a better (or faster) way to extend that list? I haven't found a function in the Python-C-API and given the overallocation of PyList_Object it seems hacky to create a new array, then memcpy the original arrays and then just assign it to ((PyList_Object *)alist)->ob_item.

tynn
  • 38,113
  • 8
  • 108
  • 143
MSeifert
  • 145,886
  • 38
  • 333
  • 352

1 Answers1

2

Your intention is implementing or using an list.extend(), but

l.extend(items)

is just equivalent to

l[len(l):] = items

This can be done within the python c api PyList_SetSlice itself, though you'll have the overhead of creating a new list. But it's better than resizing the list with every append.

static PyObject *
myfunc(PyObject *module, PyObject *alist, PyObject **items, Py_ssize_t arrsize)
{
    Py_ssize_t i, len = PyList_GET_SIZE(alist);
    PyObject *l = PyList_New(arrsize);
    int r = PyList_SetSlice(alist, len, len + arrsize, l);
    Py_CLEAR(l);
    if (r == -1) {
        return NULL;
    }

    for (i = 0 ; i < arrsize; i++) {
        Py_INCREF(items[i]);
        PyList_SET_ITEM(alist, len + i, items[i]);
    }
    return Py_INCREF(alist), alist;
}

Additionally you were loosing a reference on return.

tynn
  • 38,113
  • 8
  • 108
  • 143
  • That sounds promising, however `append` may not need to resize the list at all if the "overallocation" left enough room for the to-be-appended items. But at least your approach limits this to a maximum of one resize and one list-creation. I'll leave the question open for a bit longer before accepting, I hope you don't mind. – MSeifert Feb 12 '17 at 13:33
  • You could also reimplement the internal list resize https://github.com/python/cpython/blob/master/Objects/listobject.c#L26 – tynn Feb 12 '17 at 14:34