Numpy->Cython conversion: Compile error:Cannot convert 'npy_intp *' to Python object

Question

I have the following code that is to be propperly converted to cython:

from numpy import *

## returns winning players or [] if undecided. 
def score(board):
    scores = []
    checked = zeros(board.shape)
    for i in xrange(len(board)):
        for j in xrange(len(board)):
            if checked[i,j] == 0 and board[i,j] !=0:
                ... do stuf

my attempt at conversion to cython:

import numpy as np
cimport numpy as np

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.nonecheck(False)

## returns winning players or [] if undecided. 
def score(np.ndarray[int, ndim=2] board):
    scores = []
    cdef np.ndarray[int, ndim = 2 ] checked
    checked = np.zeros(board.shape)
    for i in xrange(len(board)):
        for j in xrange(len(board)):
            if checked[i,j] == 0 and board[i,j] !=0:
                ... do stuf

But when I compile I get :

$ python setup.py build_ext --inplace
running build_ext
cythoning newgox.pyx to newgox.c


## returns winning players or [] if undecided. 
def score(np.ndarray[int, ndim=2] board):
    scores = []
    cdef np.ndarray[int, ndim = 2 ] checked
    checked = np.zeros(board.shape)
                       ^
------------------------------------------------------------

newgox.pyx:58:28: Cannot convert 'npy_intp *' to Python object 
building 'newgox' extension

Also, I'm not sure this is the right way to work with lists in cython:

scores = []
if some_stuff_is_true:
    scores.append(some_integer)

EDIT:

Thanks, the code now compiles but when I run it I get the error:

File "newgox.pyx", line 63, in newgox.score (newgox.c:1710)
    def score(np.ndarray[np.int, ndim=2] board):
ValueError: Buffer dtype mismatch, expected 'int object' but got 'long'

I tied these two options:

ctypedef np.int_t DTYPE_t
DTYPE = np.int

and then continue to:

board = zeros((5,5), dtype = DTYPE)
def score(np.ndarray[DTYPE, ndim=2] board):

or just in both declarations:

    ...  np.int ...

This results in the same error, but with signature:

 ValueError: Buffer dtype mismatch, expected 'int' but got 'long'

---

Thanks bellamyj, but your suggested code won't compile,

$ python setup.py build_ext --inplace
running build_ext
cythoning newgox.pyx to newgox.c
building 'newgox' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c newgox.c -o   build/temp.macosx-10.6-intel-2.7/newgox.o

newgox.c:238:31: error: numpy/arrayobject.h: No such file or directory
newgox.c:239:31: error: numpy/ufuncobject.h: No such file or directory
newgox.c:356: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__pyx_t_5numpy_int8_t’
newgox.c:365: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__pyx_t_5numpy_int16_t’
newgox.c:374: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__pyx_t_5numpy_int32_t’

And it continues to enumerate all other types as well.

Then is tells me this:

newgox.c:978: error: expected ‘)’ before ‘*’ token
newgox.c:979: error: expected ‘)’ before ‘*’ token
newgox.c:980: error: expected ‘)’ before ‘*’ token
newgox.c:983: error: expected declaration specifiers or ‘...’ before ‘PyArrayObject’
newgox.c:984: error: expected declaration specifiers or ‘...’ before ‘PyArrayObject’
newgox.c:985: error: expected ‘)’ before ‘*’ token
newgox.c:987: error: ‘__pyx_t_5numpy_int32_t’ undeclared here (not in a function)
newgox.c: In function ‘__pyx_pf_6newgox_7score’:
newgox.c:1638: error: ‘PyArrayObject’ undeclared (first use in this function)
newgox.c:1638: error: (Each undeclared identifier is reported only once
newgox.c:1638: error: for each function it appears in.)
newgox.c:1638: error: ‘__pyx_v_checked’ undeclared (first use in this function)
newgox.c:1659: error: ‘__pyx_t_5’ undeclared (first use in this function)
newgox.c:1721: error: expected expression before ‘)’ token
newgox.c:1723: error: expected expression before ‘)’ token
newgox.c:1747: error: expected expression before ‘)’ token
newgox.c:1766: error: expected expression before ‘)’ token
newgox.c:1813: error: expected expression before ‘)’ token
newgox.c:1846: error: expected expression before ‘)’ token
newgox.c:1846: error: too many arguments to function ‘__pyx_f_6newgox_check_life’
newgox.c:2009: error: expected expression before ‘)’ token
newgox.c:2009: warning: assignment makes pointer from integer without a cast
newgox.c:2012: error: expected expression before ‘)’ token
newgox.c:2032: error: expected expression before ‘)’ token
newgox.c: At top level:
newgox.c:2088: error: expected declaration specifiers or ‘...’ before ‘PyArrayObject’
newgox.c: In function ‘__pyx_f_6newgox_check_life’:
newgox.c:2124: error: ‘__pyx_v_board’ undeclared (first use in this function)
newgox.c:2160: error: ‘PyArrayObject’ undeclared (first use in this function)
newgox.c:2160: error: expected expression before ‘)’ token
newgox.c:2160: error: too many arguments to function ‘__pyx_f_6newgox_liberty’
newgox.c:2420: error: too many arguments to function ‘__pyx_f_6newgox_check_life’
newgox.c: At top level:
newgox.c:2583: error: expected declaration specifiers or ‘...’ before ‘PyArrayObject’
newgox.c: In function ‘__pyx_f_6newgox_liberty’:
newgox.c:2610: error: ‘__pyx_v_board’ undeclared (first use in this function)
newgox.c: At top level:
newgox.c:2859: error: expected ‘)’ before ‘*’ token
newgox.c: In function ‘__pyx_pf_5numpy_7ndarray___getbuffer__’:
newgox.c:2999: error: ‘PyArray_Descr’ undeclared (first use in this function)
newgox.c:2999: error: ‘__pyx_v_descr’ undeclared (first use in this function)
newgox.c:3062: error: ‘PyArrayObject’ undeclared (first use in this function)
newgox.c:3062: error: expected expression before ‘)’ token
newgox.c:3071: error: ‘npy_intp’ undeclared (first use in this function)
newgox.c:3114: error: expected expression before ‘)’ token
newgox.c:3114: error: ‘NPY_C_CONTIGUOUS’ undeclared (first use in this function)
newgox.c:3154: error: expected expression before ‘)’ token
newgox.c:3154: error: ‘NPY_F_CONTIGUOUS’ undeclared (first use in this function)
newgox.c:3184: error: expected expression before ‘)’ token
newgox.c:3184: warning: assignment makes pointer from integer without a cast
newgox.c:3240: error: expected expression before ‘)’ token
newgox.c:3240: error: subscripted value is neither array nor pointer
newgox.c:3249: error: expected expression before ‘)’ token
newgox.c:3249: error: subscripted value is neither array nor pointer
newgox.c:3262: error: expected expression before ‘)’ token
newgox.c:3271: error: expected expression before ‘)’ token
newgox.c:3291: error: expected expression before ‘)’ token

And some more.

Notice it says :

newgox.c:2160: error: too many arguments to function ‘__pyx_f_6newgox_liberty’
newgox.c:2420: error: too many arguments to function ‘__pyx_f_6newgox_check_life’

But my call from score/1 to check_life is:

life, newly_checked = check_life(i,j,board,[])

And it is received like this:

# helper functions of score/1
cdef check_life(int i, int j, np.ndarray[int, ndim=2] board, checked):
    ...
    return life, checked

Finally, what kind of data type is the "i4" that you use?

Answer 1

I don't think tuple unpacking works in Cython like it does in Python. You have to specify the size of checked using (board.shape[0], board.shape[1]).

You also need to specify the datatype of checked. By default np.zeros returns a float64. You need to change that to an int32 to be compatible with the declared type.

Here's a modified version that should run.

def score(np.ndarray[int, ndim=2] board):

    cdef np.ndarray[np.int32_t, ndim = 2] checked
    checked = np.zeros((board.shape[0], board.shape[1]), dtype='i4')

    for i in xrange(len(board)):
        for j in xrange(len(board)):
            if checked[i,j] == 0 and board[i,j] !=0:
                ... do stuff

To answer the second part of your question - yes you can append to lists within Cython functions. However, as I understand it this involves Python function calls, so it won't be any faster than the same operations within Python.

Update 1:

I believe those errors are because the compiler can't find the NumPy header files. Update your setup.py to include the path to those files. The path should be output by the np.get_include() function.

import numpy as np
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
      cmdclass={'build_ext': build_ext},
      ext_modules = [
              Extension("new_gox",
                        ["new_gox.pyx"],
                        include_dirs=[np.get_include()])
                    ]
     )

The dtype i4 is a 32 bit integer. It's equivalent to np.int32.

I'm not sure what's going on with the 'too many arguments' errors. If you're calling that function from within Python, it needs to be declared as either def or cpdef. Functions declared with cdef can only be called from Cython. This is described here in the Cython docs.

Answer 2

np.intp is same as (void *)

define its type like this:

ctypedef np.npy_intp SIZE_t              # Type for indices and counters

source:

NPY_INTP == The enumeration value for a signed integer type which is the same size as a (void *) pointer. This is the type used by all arrays of indices.

So for example in cython...

Struct in header file.h file:

struct Cudamat_Struct1 {
    float * dataHost;
    float * dataDevice;
    int onDevice;
    int onHost;
    (void *) shape;
    int size;
    int isTrans; // 0 or 1
    int ownsData;

};

In cython file.pxd file:

cdef extern from "file.h":
    cdef cppclass c_Cudamat_Struct "Cudamat_Struct":
        float *dataHost
        float *dataDevice
        bint onDevice
        bint onHost
        SIZE_t * shape
        int size
        bint isTrans
        bint ownsData

Finally in cython function:

cdef pycublas_thrust_2d():
    cdef c_Cudamat_Struct1* mat = new c_Cudamat_Struct1()

    cdef int N = 10

    cdef np.ndarray[ndim=2, dtype=F32t] arrHost = np.ones((1,10),dtype=F32)

    cdef int size = 10

    cdef F32t[:,:] arrHost_ = arrHost

    cdef F32t * arrHostPtr = &arrHost[0,0]

    mat.dataHost = arrHostPtr
    mat.shape = arrHost.shape
    mat.size = size