Cython overhead on extension types in memoryview

Question

I am compiling a Cython module, and checked this piece of code with cython -a command.

cdef INT_t print_info(Charge[:] electrons):
    cdef INT_t i, index
    for i in range(electrons.shape[0]):
        index = electrons[i].particleindex
    return index

It turns out that

 + index = electrons[i].particleindex
    __pyx_t_4 = __pyx_v_i;
    __pyx_t_3 = (PyObject *) *((struct __pyx_obj_14particle_class_Charge * *) ( /* dim=0 */ (__pyx_v_electrons.data + __pyx_t_4 * __pyx_v_electrons.strides[0]) ));
    __Pyx_INCREF((PyObject*)__pyx_t_3);
    __pyx_t_5 = ((struct __pyx_obj_14particle_class_Charge *)__pyx_t_3)->particleindex;
    __Pyx_DECREF(((PyObject *)__pyx_t_3)); __pyx_t_3 = 0;
    __pyx_v_index = __pyx_t_5;

Charge is a cdef extension type and I am trying to use a memoryview buffer Charge[:] here. It seems that Cython calls some Python API in this case, in particular __Pyx_INCREF((PyObject*) and __Pyx_DECREF(((PyObject *) have been generated.

I am wondering what causes this, will it cause a lot of slowdown? It is my first post in the forum, any comments or suggestions are greatly appreciated!

PS: Charge object is defined as

charge.pyx

cdef class Charge:
    def __cinit__(Charge self):
        self.particleindex = 0     
        self.charge = 0
        self.mass = 0
        self.energy = 0
        self.on_electrode = False

charge.pxd

cdef class Charge:
    cdef INT_t particleindex
    cdef FLOAT_t charge
    cdef FLOAT_t mass
    cdef FLOAT_t energy
    cdef bint on_electrode

`INCREF`,`DECREF` - increase/decrease reference (count). This is part of keeping track of references to Python objects (for garbage collection). `_pyx_t_3` is a new (temporary) object of class `Charge`, created by the indexing. — hpaulj, Dec 06 '15 at 02:44
Thank you for your reply and editing of my thread. Is there anything wrong with the definition? — Luman, Dec 06 '15 at 04:58
So by indexing `electrons[i]`, you are creating a temporary reference to a `Charge` object. That's what `INCREF` notes. Once the `partaicleindex` is 'stored' in `pyx_t_5`, it throws `_pyx_t_3` away, and decreases the reference count. `INCREF/DEC` are just normal Python bookkeeping. — hpaulj, Dec 06 '15 at 05:34
Thanks for the explanation. Does this procedure have any penalty on the speed of code? If so, is there anyway to overcome it? — Luman, Dec 06 '15 at 07:00
Don't use lists/arrays of extension types or objects where speed is critical. — hpaulj, Dec 06 '15 at 08:19

score 0 · Answer 1 · answered Dec 05 '15 at 23:34

0

Cython will likely be happier with pythonic code. Rewrite your function:

cdef INT_t print_info(Charge[:] electrons):
    cdef INT_t i, index
    for electron in electrons:
        index = electron.particleindex
    return index

and try again.

answered Dec 05 '15 at 23:34

Mike Müller

82,630
20
166
161

Thank you for your reply. I tried but it does not work. The line gets even more 'yellow'. Also I think it has something to do with the memoryview buffer. When I just use the extension type as argument, say some_function(Charge electron), Cython does not generate the API layer. – Luman Dec 06 '15 at 01:39

score 0 · Answer 2 · answered Jan 04 '16 at 03:53

It's not the memoryview, it's the extension type. Cython extension types are treated with the same reference-counting semantics as Python objects.

You can get and work with pointers to them which do not change recounts, either with <void*> or with <PyObject*> (which you can cimport from cpython.ref), but the pointers obviously don't have any methods or attributes. The minute you try to cast back to the extension type type, the INCREF/DECREF code reappears. Those instructions are pretty fast though.

There was some talk on the mailing list about how non-refcounted references (i.e., with access to object data) to extension types might be a new feature in the future, but there seemed to be little enthusiasm for adding a feature that, realistically, is probably going to lead to a lot of horrifyingly buggy code of the "access violation" variety.

Cython overhead on extension types in memoryview

2 Answers2