Can we create a C-array of Python objects in Cython?

Question

Let's consider the following code:

class C:                                                                                                                                                                                            
    pass                                                                                                                                                                                            
                                                                                                                                                                                                    
cdef object f():                                                                                                                                                                                    
    return C()

cdef void g(unsigned n):
    cdef object obj0 = f()
    cdef object obj1 = f()
    cdef object obj2 = f()
    cdef object obj3 = f()

Is there a way to store the various objects in an array instead of using several variables? Something along the lines of:

cdef void g(unsigned n):
    cdef object[N_MAX] obj
    for i in range(n)
       obj[i] = f()

This is very very close to a duplicate of https://stackoverflow.com/questions/33851333/cython-how-do-you-create-an-array-of-cdef-class/33853634#33853634. You want a `list`. — DavidW, Apr 11 '23 at 19:11
Thanks, David. I saw the discussion at https://groups.google.com/g/cython-users/c/G8zLWrA-lU0 -- but I missed your previous answer. I look at that now. — Sylvain Leroux, Apr 11 '23 at 19:14
The answer is more to do with an array of *specific* object type (which is why it isn't quite a duplicate) so some of the suggestions don't apply. But a list (or maybe a tuple, if you don't need to reassign) is probably the best solution - internally it's surprisingly close to a plain array of object pointers. — DavidW, Apr 11 '23 at 19:18
Yes. I didn't have the time to try that this evening -- but I will surely use a list. Maybe using directly [PyList_New](https://docs.python.org/3/c-api/list.html#c.PyList_New) to avoid the burden of creating the list incrementally by calling `append`. — Sylvain Leroux, Apr 11 '23 at 21:02
Hi, @DavidW. I made several tests, and in the end, the simplest is certainly the better. I will use a list comprehension to store the array of objects. I posted my solution as an answer: https://stackoverflow.com/a/75998181/2363712 — Sylvain Leroux, Apr 12 '23 at 17:17

Sylvain Leroux · Answer 1 · 2023-04-12T17:34:11.663

It is not really possible to create an array of object. One caveat is, if it would, we should manually manage the lifetime of the objects using Py_INCREf/Py_DECREF.

The simplest and safest move is probably to maintain the list of objects at the Python layer. Following the example given in the original question, we could, for example, use a list comprehension:

cdef void g(unsigned n):                                                                                                                                                                            
    cdef list l = [f() for _ in range(n)]

    #...

This is compiled into something along the lines of:

  __pyx_t_1 = PyList_New(0); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 10, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  __pyx_t_2 = __pyx_v_n;
  for (__pyx_t_3 = 0; __pyx_t_3 < __pyx_t_2; __pyx_t_3+=1) {
    __pyx_v__ = __pyx_t_3;
    __pyx_t_4 = __pyx_f_1a_f(); if (unlikely(!__pyx_t_4)) __PYX_ERR(0, 10, __pyx_L1_error)
    __Pyx_GOTREF(__pyx_t_4);
    if (unlikely(__Pyx_ListComp_Append(__pyx_t_1, (PyObject*)__pyx_t_4))) __PYX_ERR(0, 10, __pyx_L1_error)
    __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
  }
  __pyx_v_l = ((PyObject*)__pyx_t_1);
  __pyx_t_1 = 0;

I was tempted to manually manage the list using PyList_New and PyList_SET_ITEM to avoid the successive calls to __Pyx_ListComp_Append:

from cpython cimport PyList_New, PyList_SET_ITEM, Py_INCREF
cdef void h(unsigned n):
    cdef list l = PyList_New(n)
    cdef unsigned i
    for i in range(n):
        o = f()
        Py_INCREF(o)
        PyList_SET_ITEM(l, i, o)

    #...

In practice, that makes the code much less readable (and more error-prone), for very little benefits.

Can we create a C-array of Python objects in Cython?

1 Answers1