How to `pop` element from PyListObject?

Question

Lets say I have a PyListObject, and I want to append a PyObject then I can use PyList_Append API which is documented in List Objects C-API. But for my use case I want to pop an element from the PyListObject(i.e. my_list.pop() in python layer).

But the List Objects C-API documentation does not mention anything about pop operation.

So my question is, How can I pop an element from PyListObject using C-API.

Answer 1

No, the list.pop method is not directly available via the C-API on PyListObjects.

Given that list.pop already exists and is implemented in C you could simply look up what the CPython implementation does:

static PyObject *
list_pop_impl(PyListObject *self, Py_ssize_t index)
{
    PyObject *v;
    int status;

    if (Py_SIZE(self) == 0) {
        /* Special-case most common failure cause */
        PyErr_SetString(PyExc_IndexError, "pop from empty list");
        return NULL;
    }
    if (index < 0)
        index += Py_SIZE(self);
    if (index < 0 || index >= Py_SIZE(self)) {
        PyErr_SetString(PyExc_IndexError, "pop index out of range");
        return NULL;
    }
    v = self->ob_item[index];
    if (index == Py_SIZE(self) - 1) {
        status = list_resize(self, Py_SIZE(self) - 1);
        if (status >= 0)
            return v; /* and v now owns the reference the list had */
        else
            return NULL;
    }
    Py_INCREF(v);
    status = list_ass_slice(self, index, index+1, (PyObject *)NULL);
    if (status < 0) {
        Py_DECREF(v);
        return NULL;
    }
    return v;
}

Source for CPython 3.7.2

This includes a lot of functions that are not (easily) accessible for a C extension and it also handles popping from a specific index (even negative ones). Personally I wouldn't even bother to re-implement it but just call the pop method with PyObject_CallMethod:

PyObject *
list_pop(PyObject *lst){
    return PyObject_CallMethod(lst, "pop", "n", Py_SIZE(lst) - 1);
}

It might be a bit slower than a re-implementation but it should be "safer" - one cannot accidentally mess up invariants of the list object (for example resize conditions).

Another implementation is present in Cython

static CYTHON_INLINE PyObject* __Pyx_PyList_Pop(PyObject* L) {
    /* Check that both the size is positive and no reallocation shrinking needs to be done. */
    if (likely(PyList_GET_SIZE(L) > (((PyListObject*)L)->allocated >> 1))) {
        Py_SIZE(L) -= 1;
        return PyList_GET_ITEM(L, PyList_GET_SIZE(L));
    }
    return CALL_UNBOUND_METHOD(PyList_Type, "pop", L);
}

That could also be adapted for your use-case.

Answer 2

You will have to roll it out by yourself. Here is a possible implementation (without error checking):

PyObject *my_pop_from_list(PyListObject *lst){
    //TODO: check lst isn't empty
    Py_SIZE(lst) -= 1;                                 // forget last element 
    return PyList_GET_ITEM(lst, PyList_GET_SIZE(lst)); // return last element
}

Py_SIZE is just a macro to access lst->ob_size, which we decrease while executing pop.

Also versions without error checking, i.e. PyList_GET_ITEM and PyList_GET_SIZE, are used, because once it is established (see TODO-comment), that the list isn't empty - nothing could go wrong.

The caller receives a new reference, albeit PyList_GET_ITEM returns a borrowed one: decreasing the size of the list the way we did in the code above, makes list "forget" the reference without decreasing the reference counter.

As @MSeifert has pointed out, this version doesn't change the size of the underlying array, the same way list.pop() would do (if only half or less of the underlying array is used after pop). This could be seen as "feature" of the above implementation - trading speed for memory.