3

I'm trying to utilize the concepts in this sample code to run some Cython code in parallel, but I can't seem to find any information in the Cython documentation about what this notation actually means.

    cdef FLOAT_t[:] numbers
    cdef unsigned int i
    cdef INDEX_t n_workers
    cdef PyObject **workers
    cdef list ref_workers #Here to maintain references on Python side
    
    def __init__(Parent self, INDEX_t n_workers, list numbers):
        cdef INDEX_t i
        self.n_workers = n_workers
        self.numbers = np.array(numbers,dtype=float)
        self.workers = <PyObject **>malloc(self.n_workers*cython.sizeof(cython.pointer(PyObject)))
        
        #Populate worker pool
        self.ref_workers = []
        for i in range(self.n_workers):
            self.ref_workers.append(Worker())
            self.workers[i] = <PyObject*>self.ref_workers[i]
    
    def __dealloc__(Parent self):
        free(self.workers)

Does the ** notation mean that it is a pointer to a pointer of a PyObject? I understand that the <> notation is meant to dereference the pointer, so is this line:

self.workers = <PyObject **>malloc(self.n_workers*cython.sizeof(cython.pointer(PyObject)))

allocating an unknown amount of memory, since the size of the PyObject is unknown until self.workers is filled with dereferenced PyObjects?

stagermane
  • 1,003
  • 2
  • 12
  • 29

1 Answers1

1

Not only is it a pointer to a PyObject* pointer it's also the pointer to the first element of an array of PyObject* pointers.

You can see it is allocating memory to accommodate self.n_workers, presumably the workers are implemented using a PyObject derivative, so in memory you will have:

self.workers -> self.workers[0] (PyObject* for 1st worker)
                self.workers[1] (PyObject* for 2nd worker)
                ....
                self.workers[N-1]  (PyObject* for last worker)
Den-Jason
  • 2,395
  • 1
  • 22
  • 17
  • 1
    I think another issue of the OP is, that they confuse dereferece-operator (`*`) with cast of `void *` to `PyObject **` via ``. – ead Sep 05 '20 at 18:51
  • Ah yes, you're right that ```<>``` indicates casting rather than dereferencing, which is done by ```[0]``` syntax (though I don't know how that would work with a pointer to a pointer). Ultimately, I'm trying to adapt this code for use with a similar problem to take advantage of prange (since the parent class is able to use worker class objects), but this sample code actually takes longer to run in parallel than it does in serial! – stagermane Sep 06 '20 at 06:00
  • @stagermane if you're trying to run Python code in parallel via Cython you might be out of luck; I believe the GIL prevents multiprocessing from occurring. – Den-Jason Sep 06 '20 at 23:23
  • 1
    @Den-Jason, was able to fix this, but you can run in parallel with Cython by releasing the GIL, your function just has to be free of python objects. – stagermane Sep 08 '20 at 00:59