0

I am currently using the new NUMPY_1_7 C API and Cython 0.29+. Usage of types like cnp.ndarray are deprecated and it is preferred to use a Cython memoryview instead.

However, part of my code stores a ndarray of type 'object'. E.g.

cdef class A:
   cdef cnp.ndarray buff

   cdef void A_func(self): 
        self.buff = np.empty(10, dtype='object')
        
        cdef int idx
        for idx in range(10):
             self.buff[idx] = CythonClass(5)

   cdef void nogil_func(self) nogil:
        cdef void** buff = <void**> self.buff
        cdef int idx

        for idx in range(10):
             (< CythonClass > buff[idx]).do_something_nogil()

cdef class CythonClass:
    def __cinit__(self, int n):
        self.n = n

    cdef int do_something_nogil(self) nogil:
        cdef int result = 0
        cdef int idx

        for idx in range(5):
             result += idx
        return result

I have a few questions of how to covert the cdef cnp.ndarray buff line into a Cython memoryview, such that it allows me to still leverage the data structure in nogil blocks:

  1. How would I go about converting buff into a Cython memorvyew that supports the storage of an "object"? Is the way below fine?
  2. Are there alternatives?
  3. Are there tradeoffs to the alternatives in terms of speed and efficiency?

Here is my attempt:

cdef class A:
    cdef object[:] buff

   cdef void A_func(self): 
        self.buff = np.empty(10, dtype='object')
        
        cdef int idx
        for idx in range(10):
             self.buff[idx] = CythonClass(5)

   cdef void nogil_func(self) nogil:
        cdef void** buff = <void**> self.buff
        cdef int idx

        for idx in range(10):
             (<CythonClass> buff[idx]).do_something_nogil()

which results in the following error: operand of type '__Pyx_memviewslice' where arithmetic or pointer type is required

ajl123
  • 1,172
  • 5
  • 17
  • 40
  • `np.empty(10, dtype='object')` - At least in Python, this array will be filled with `None`. You'll have to call `CythonClass(5)` for each (distinct) instance that you want to put in the array. Unless you want all elements to reference the same instance, as `[cythonClass(5)]*10` would do. – hpaulj Feb 07 '23 at 16:41
  • At least at the Python level, there's nothing efficient about object dtype arrays. They are like lists. – hpaulj Feb 07 '23 at 17:12
  • I think this is basically a duplicate of https://stackoverflow.com/q/33851333/4657412 – DavidW Feb 07 '23 at 18:20
  • Is it my understanding then the best “cythonic” way to do it is define a memory view of “object”? (‘object [:]’)? Since I am in a Cython function I would prefer not using a Python list for this. But perhaps for performance, it seems it would be worth investigating the usage of the C structs approach or numpy structured array? A numpy array of type object wouldn’t work because IIUC using cnp.ndarrays are deprecated. – ajl123 Feb 07 '23 at 20:19
  • A `list` (typed as `list`) should have very similar performance to a 1D memoryview of `object[:]` so pick what is more convenient. Just typing something as `cnp.ndarray` will probably be worse performance since Cython doesn't know the underlying dtype (so can't optimize much). C structs or structured arrays are potentially better performance but require adapting other code. – DavidW Feb 07 '23 at 20:28
  • Ah thank you for the explanation @DavidW. I think perhaps I was asking the wrong line of questioning earlier. I updated my post to add more details. Beyond just converting from `cnp.ndarray` to a non-deprecated data structure (i.e. `object` memoryview), I want to continue supporting downstream functionality of nogil functions. So this might be slightly different from the other post (I think). Thank you for wading thru this rabbit hole... – ajl123 Feb 07 '23 at 21:55
  • You can do a `CythonClass[:]` memoryview instead of an `object[:]` memoryview to avoid the casts. It doesn't validate the element types (so it may give you a false sense of security), but it lets you access `cdef` methods directly. I don't know how/if that works with `nogil` though off the top of my head – DavidW Feb 07 '23 at 22:07

0 Answers0