Cython: Invalid use of fused types, type cannot be specialized

Question

I have the following MCVE:

import numpy as np

cimport numpy as np
cimport cython

from cython cimport floating


def func1(floating[:] X_data, floating alpha):
    if floating is double:
        dtype = np.float64
    else:
        dtype = np.float32

    cdef floating[:] prios = np.empty(12, dtype=dtype)
    cdef int ws_size = 10

    C = np.argpartition(np.asarray(prios), ws_size)[:ws_size].astype(np.int32)

    cdef int res = func2(X_data, alpha, C)


cpdef int func2(floating[:] X_data, floating alpha, int[:] C):
    cdef int epoch = 1
    return epoch

Trying to run cython test_fused.pyx gives me:

Error compiling Cython file:
------------------------------------------------------------
...
    cdef floating[:] prios = np.empty(12, dtype=dtype)
    cdef int ws_size = 10

    C = np.argpartition(np.asarray(prios), ws_size)[:ws_size].astype(np.int32)

    cdef int res = func2(X_data, alpha, C)
                       ^
------------------------------------------------------------

test_fused.pyx:21:24: no suitable method found

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
        cdef floating[:] prios = np.empty(12, dtype=dtype)
        cdef int ws_size = 10

        C = np.argpartition(np.asarray(prios), ws_size)[:ws_size].astype(np.int32)

        cdef int res = func2(X_data, alpha, C)
                           ^
    ------------------------------------------------------------

    test_fused.pyx:21:24: no suitable method found

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
        cdef floating[:] prios = np.empty(12, dtype=dtype)
        cdef int ws_size = 10

        C = np.argpartition(np.asarray(prios), ws_size)[:ws_size].astype(np.int32)

        cdef int res = func2(X_data, alpha, C)
                      ^
    ------------------------------------------------------------

    test_fused.pyx:21:19: Invalid use of fused types, type cannot be specialized

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
        cdef floating[:] prios = np.empty(12, dtype=dtype)
        cdef int ws_size = 10

        C = np.argpartition(np.asarray(prios), ws_size)[:ws_size].astype(np.int32)

        cdef int res = func2(X_data, alpha, C)
                      ^
    ------------------------------------------------------------

    test_fused.pyx:21:19: Invalid use of fused types, type cannot be specialized

I had a more complicated code which also passed the array C as a runtime defined value, which did not cause any issue. What is the cause of this compilation error?

I am puzzled, because slight modifications (adding a dummy keyword arg to func1 and two keyword args to func2) make the code compile:

def func1(floating[:] X_data, floating alpha,
          int dummy_variable=1):  # added dummy_variable here

    # same as before here

    cdef int res = func2(X_data, alpha, C,
                       dummy_variable=dummy_variable)


cpdef int func2(floating[:] X_data, floating alpha, int[:] C, 
    int K=6, int dummy_variable=1):  # added K and dummy variable here

    cdef int epoch = 1
    return epoch

Answer 1

Let's start with a smaller reproducer:

%%cython
import numpy as np
from cython cimport floating

def func1(floating[:] X_data):
    C = np.empty(12, dtype=np.int_32)
    func2(X_data, C)

cpdef func2(floating[:] X_data, int[:] C):
    pass

It doesn't compile.

One important observation: func2 is a cpdef, that means Cython will call it as a raw-C-function from func1. These two C-signatures will be produced by Cython for fused func2 function (one for double and one for float):

static PyObject *__pyx_fuse_0__pyx_f_4test_func2(__Pyx_memviewslice, __Pyx_memviewslice, int __pyx_skip_dispatch); /*proto*/
static PyObject *__pyx_fuse_1__pyx_f_4test_func2(__Pyx_memviewslice, __Pyx_memviewslice, int __pyx_skip_dispatch); /*proto*/

So C is expected to be a __Pyx_memviewslice, but as long as Cython is concerned it is a PyObject in func1, so there is no way the function can be called as cdef. What I don't understand: why doesn't Cython fall back to python-def version?

The C signature is a little bit misleading and Cython does more type checking during the compilation, so it would not help to define C as

cdef float[:] C

because even if C is in this case also a __Pyx_memviewslice it doesn't have the right type, and only

cdef int[:] C

would work.

If func2 were defined as

cpdef func2(floating[:] X_data, C):

the corresponding two C-signatures would be

static PyObject *__pyx_fuse_0__pyx_f_4test_func2(__Pyx_memviewslice, PyObject *, int __pyx_skip_dispatch); /*proto*/
static PyObject *__pyx_fuse_1__pyx_f_4test_func2(__Pyx_memviewslice, PyObject *, int __pyx_skip_dispatch); /*proto*/

so it is possible to pass C, which is a PyObject to these function.

So there are twos way to solve the compilation issue:

• use cdef int[:] C in func1, or
• drop int[:] C in the signature of func2

---

So why does adding a dummy-parameter, i.e.

%%cython -a
import numpy as np
from cython cimport floating

def func1(floating[:] X_data, int dummy_variable=1):
    C = np.empty(12, dtype=np.int_32)
    func2(X_data, C, dummy_variable=dummy_variable)

cpdef func2(floating[:] X_data, int[:] C, int k=6, dummy_variable = 1):
    pass

works?

Actually there is a third way to make the code compile: by making func2 a python-only def-function. In this case, the type of C doesn't play a role during the compile time and will be checked at run time.

The thing is: the case with your dummy-variables, Cython decides to call func2 as a Python-function and not as C-function, and thus the type mismatch doesn't play a role.

You can easily see that by inspecting the annotated html-file.

However, I cannot say, what is the reason for Cython to fall back to a Python-function-call for your work-around. All I can tell: don't providing a value for k plays a important role.