Iterating over a list in parallel with Cython

Question

How does one iterate in parallel over a (Python) list in Cython?

Consider the following simple function:

def sumList():
    cdef int n = 1000
    cdef int sum = 0

    ls = [i for i in range(n)]

    cdef Py_ssize_t i
    for i in prange(n, nogil=True):
        sum += ls[i]

    return sum

This gives a lot of compiler errors, because a parallel section without the GIL apparently cannot work with any Python object:

Error compiling Cython file:
------------------------------------------------------------
...

    ls = [i for i in range(n)]

    cdef Py_ssize_t i
    for i in prange(n, nogil=True):
        sum += ls[i]
     ^
------------------------------------------------------------

src/parallel.pyx:42:6: Coercion from Python not allowed without the GIL

Error compiling Cython file:
------------------------------------------------------------
...

    ls = [i for i in range(n)]

    cdef Py_ssize_t i
    for i in prange(n, nogil=True):
        sum += ls[i]
     ^
------------------------------------------------------------

src/parallel.pyx:42:6: Operation not allowed without gil

Error compiling Cython file:
------------------------------------------------------------
...

    ls = [i for i in range(n)]

    cdef Py_ssize_t i
    for i in prange(n, nogil=True):
        sum += ls[i]
     ^
------------------------------------------------------------

src/parallel.pyx:42:6: Converting to Python object not allowed without gil

Error compiling Cython file:
------------------------------------------------------------
...

    ls = [i for i in range(n)]

    cdef Py_ssize_t i
    for i in prange(n, nogil=True):
        sum += ls[i]
          ^
------------------------------------------------------------

src/parallel.pyx:42:11: Indexing Python object not allowed without gil

This might not be immediately useful, but have you ever tried D? It's c syntax and stuff like parallel iteration over lists (arrays or ranges in D) is almost as easy as doing it in MATLAB, if you're familiar. Check out the example here: [std.parallelism](http://dlang.org/phobos/std_parallelism). I've found it very easy to come to from a Python background. — Matthew Turner, Jul 23 '13 at 20:54
@mattyTpain Experimented with D, liked the pythonic syntax. But here is an issue I ran into: http://stackoverflow.com/questions/17837098/parallel-iterators-in-the-d-language — clstaudt, Jul 24 '13 at 14:35

score 8 · Accepted Answer · edited Apr 03 '14 at 13:18

I am not aware of any way to do this. A list is a Python object, so using its __getitem__ method requires the GIL. If you are able to use a NumPy array in this case, it will work. For example, if you wanted to iterate over an array A of double precision floating point values you could do something like this:

cimport cython
from numpy cimport ndarray as ar
from cython.parallel import prange
@cython.boundscheck(False)
@cython.wraparound(False)
cpdef cysumpar(ar[double] A):
    cdef double tot=0.
    cdef int i, n=A.size
    for i in prange(n, nogil=True):
        tot += A[i]
    return tot

On my machine, for this particular case, prange doesn't make it any faster than a normal loop, but it could work better in other cases. For more on how to use prange see the documentation at http://docs.cython.org/src/userguide/parallelism.html

Whether or not you can use an array depends on how much you are changing the size of the array. If you need a lot of flexibility with the size, the array will not work. You could also try interfacing with the vector class in C++. I've never done that myself, but there is a brief description of how to do that here: http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#nested-class-declarations

On the other hand, if you only need a sum, you may as well just use the `sum` method of NumPy arrays. — IanH, Jul 23 '13 at 20:42
Thanks for the hints. The sum is just a simple example, I need more complex operations applied in parallel. — clstaudt, Jul 24 '13 at 08:33
Right, anything that involves using the Python interpreter can't be used in a nogil context since the interpreter isn't thread safe. That applies to most things in NumPy. Cython can remove the Python calls in cases like this for single-item-access though and then you get the equivalent of a C loop. — IanH, Oct 02 '17 at 22:01

score 1 · Answer 2 · answered May 22 '15 at 18:05

Convert your list into an array if you need any numeric value, or a bytearray if values are limited between 0 and 255. If you store anything else than numeric values, try numpy or use dtypes directly. For example with bytes:

cdef int[::1] gen = array.array('i',[1, 2, 3, 4])

And if you want to use C types:

ctypedef unsigned char uint8_t

Iterating over a list in parallel with Cython

2 Answers2