Shared memory with Cython multiprocessing

Question

I am trying to define an Array in shared memory with Cython and the multiprocessing package. However, I am not able to declare this array in the .pxd file.

The simple test code is as follows. I have a Cython class Data with the variable my_data. This variable my_data will be the shared array.

Data.pxd:

    cimport numpy as np
    cdef class Data:
        cdef public np.ndarray my_data

Data.pyx:

    cdef class Data:
        def __init__(self):
            pass

Then, I declare the shared array in my main file:

main_file.py:

    # -*- coding: utf-8 -*-
    import pyximport; 
    import numpy as np
    from multiprocessing import Array
    pyximport.install(setup_args={"include_dirs":np.get_include()},)

    from Data import *

    ### Create Data
    N  = 1500
    dc = Data()
    dc.my_data  = Array('d', N, lock=False)

Running main_file.py gives the TypeError:

    TypeError: Cannot convert c_double_Array_1500 to numpy.ndarray

I tried to declare my_data as a cpython array, but that gives the same TypeError. Is there a way to declare my_data in Data.pxd such that it can be shared with the multiprocesssing?

score 0 · Accepted Answer · answered Aug 12 '15 at 07:13

The problem you're having is that the multiprocessing.Array isn't a numpy array but is instead a "ctypes array" (according to the multiprocessing documentation).

The easiest way of making it work is to use the Cython typed memoryview feature, which accepts anything that has an array interface:

# Data.pxd
cdef class Data:
    cdef public double[:] my_data

When I make this change your program works fine, and will also work fine with numpy arrays, python arrays, and most other things you try to throw at it.

Note that the one thing that is inflexible in this version is that the array is an array of doubles, which you didn't specify. If you need to accept any datatype here (and not just doubles) you probably can't do it with a cdef type easily.

Shared memory with Cython multiprocessing

1 Answers1