Numba and Cython aren't improving the performance compared to CPython significantly, maybe I am using it incorrectly?

Question

BIG EDIT:

================

For the sake of clarity, I am removing the old results and replace it by the more recent results. The question is still the same: Am I using both Cython and Numba correctly, and what improvements to the code can be made? (I have a newer and more bare-bones temporary IPython notebook with all the code and results here)

1)

I think I figured out why there was initially no difference between Cython, Numba, and CPython: It was because I fed them

numpy arrays as input:

x = np.asarray([x_i*np.random.randint(8,12)/10 for x_i in range(n)])

instead of lists:

x = [x_i*random.randint(8,12)/10 for x_i in range(n)]

Benchmark using Numpy arrays as data input

Benchmark using Python lists as input

2)

I replaced the zip() function by explicit loops, however, it didn't make much of a difference. The code would be:

CPython

def py_lstsqr(x, y):
    """ Computes the least-squares solution to a linear matrix equation. """
    len_x = len(x)
    x_avg = sum(x)/len_x
    y_avg = sum(y)/len(y)
    var_x = 0
    cov_xy = 0
    for i in range(len_x):
        temp = (x[i] - x_avg)
        var_x += temp**2
        cov_xy += temp*(y[i] - y_avg)
    slope = cov_xy / var_x
    y_interc = y_avg - slope*x_avg
    return (slope, y_interc) 

Cython

%load_ext cythonmagic

%%cython
def cy_lstsqr(x, y):
    """ Computes the least-squares solution to a linear matrix equation. """
    cdef double x_avg, y_avg, var_x, cov_xy,\
         slope, y_interc, x_i, y_i
    cdef int len_x
    len_x = len(x)
    x_avg = sum(x)/len_x
    y_avg = sum(y)/len(y)
    var_x = 0
    cov_xy = 0
    for i in range(len_x):
        temp = (x[i] - x_avg)
        var_x += temp**2
        cov_xy += temp*(y[i] - y_avg)
    slope = cov_xy / var_x
    y_interc = y_avg - slope*x_avg
    return (slope, y_interc)

Numba

from numba import jit

@jit
def numba_lstsqr(x, y):
    """ Computes the least-squares solution to a linear matrix equation. """
    len_x = len(x)
    x_avg = sum(x)/len_x
    y_avg = sum(y)/len(y)
    var_x = 0
    cov_xy = 0
    for i in range(len_x):
        temp = (x[i] - x_avg)
        var_x += temp**2
        cov_xy += temp*(y[i] - y_avg)
    slope = cov_xy / var_x
    y_interc = y_avg - slope*x_avg
    return (slope, y_interc)

Answer 1

Here's what I think is happening with Numba:

Numba works on Numpy arrays. Nothing else. Everything else has nothing to do with Numba.

zip returns an iterator of arbitrary items, which Numba cannot see into. Thus Numba cannot do much compiling.

Looping over the indexes with a for i in range(...) is likely to produce a much better result and allow much stronger type inference.

Answer 2

Using the builtin sum() could be causing problems.

Here's linear regression code that will run faster in Numba:

@numba.jit
def ols(x, y):
    """Simple OLS for two data sets."""
    M = x.size

    x_sum = 0.
    y_sum = 0.
    x_sq_sum = 0.
    x_y_sum = 0.

    for i in range(M):
        x_sum += x[i]
        y_sum += y[i]
        x_sq_sum += x[i] ** 2
        x_y_sum += x[i] * y[i]

    slope = (M * x_y_sum - x_sum * y_sum) / (M * x_sq_sum - x_sum**2)
    intercept = (y_sum - slope * x_sum) / M

    return slope, intercept