2

I would like to use in a cython code a C++ std::priority_queue where each item should be a struct or a class, but I don't know how to define a comparator.

If I consider a priority_queue which elements are standard types everything seems ok as in the following example:

simple.pyx

# distutils: language = c++
cimport cython
from libcpp.queue cimport priority_queue

def testIntPriorityQueue():
    cdef priority_queue[int] q
    q.push(5)
    q.push(15)
    q.push(7)

    print('q.top (after pushing 5, 15, 7) = ',q.top())

    q.pop()
    print('q.top (after popping one item) = ',q.top())

main.py

import simple 

def main():
    simple.testIntPriorityQueue()
   
main()

When I execute this code the priority_queue, after pushing 5, 15 and 7 correctly has on top the greatest (15) (as it's returned by q.top) and after popping out one elements q.top returns 7 that has become the greatest after the removal of 15.

Now I would like to have a similar behaviour but with items pushed into the queue that are not standard type objects but something more complex.

I've tried defining such a struct with the following things:

simple.pxd

cimport cython
ctypedef struct myType:
    int key
    float a
    float b

simple.pyx

# distutils: language = c++
cimport cython
from libc.stdlib cimport  malloc, free
from libcpp.queue cimport priority_queue
from libcpp.vector cimport vector

def testPriorityQueue():
    cdef myType * item
    
    cdef priority_queue[myType*] q

    item = <myType *> malloc(sizeof(myType)) 
    item.key = 20
    item.a = 3.
    item.b = 4.
    q.push(item)

    item = <myType *> malloc(sizeof(myType)) 
    item.key = 10
    item.a = 1.
    item.b = 2.
    q.push(item)
    
    item = q.top()
    print('q.top',item.key,item.a,item.b)

Now no error is provided neither from the compiler nor at runtime, but of course the ordering in the priority_queue cannot be defined and from the prints I get that the item on top is the last pushed.

In C++ to make priority_queue to sort pushed items in a such situation a comparison method should be provided as an argument to the queue instantiation, but this, as from what I've found, should be done through an overloading of the less operator as in this C++ example:

struct ToastCompare
{
    bool operator()(const Toast &t1, const Toast &t2) const
    {
        int t1value = t1.bread * 1000 + t1.butter;
        int t2value = t2.bread * 1000 + t2.butter;
        return t1value < t2value;
    }
};

I've tried to define a similar function in cython by adding these lines to pyx file:

simple.pyx

cdef extern from *:
    """
    struct itemCompare{
        bool operator() (self, myType *t1, myType *t2) const
        {
            return t1.key < t2.key;
        }
    };
    """
    ctypedef struct itemCompare
cdef priority_queue[myType*,vector[myType*],itemCompare] q

But during compiling I get a "priority_queue templated type receives 1 arguments, got 3" error.

Is it possible in cython to use C++ priority_queue container with custom class objects?


Trying to follow @ead suggestion, I've tried to modify the code in this way:

simple.pyx

cdef extern from *:
    """
    bool operator<(const myType t1, const myType t2)
    { return t1.key < t2.key;}
    """
def testPriorityQueue():
    cdef priority_queue[myType*] q

leaving in simple.pxd

cimport cython
from libcpp cimport bool
ctypedef struct myType:
    int key
    float a
    float b

but it cannot compile giving:

simple.cpp:694:26: error: ‘myType’ does not name a type bool operator<(const myType t1, const myType t2)

So I tried moving struct definition in the C-verbatim code and removing it from simple.pxd

simple.pyx

def extern from *:
    """
    typedef struct myType{
        int key;
        float a;
        float b;       
    } myType;

    bool operator<(const myType t1, const myType t2)
    {return t1.key < t2.key;}
    """
    ctypedef struct myType
def testPriorityQueue():
    cdef myType * item
    cdef priority_queue[myType*] q

    item = <myType *> malloc(sizeof(myType)) 

and now the compiling error is:

Cannot take sizeof incomplete type 'myType'

If I leave struct definition also in the pxd file but removing ctypedef struct myType from pyx no compiling error occurs, but the priority queue doesn't sort items as if no overloading has been done.

Where is my error?

Daniel Farrell
  • 9,316
  • 8
  • 39
  • 62
  • It will take time for this question to get opened - it is always better to get the question right from the beginning. – ead Apr 23 '20 at 21:11
  • In the meantime there are some misunderstandings: 1) you need to override operator less (https://stackoverflow.com/q/3006413/5769463) not operator(). 2) put your objects themselves and not pointers into the priority_queue(if the objects are as small as you have presented) - no problems with memory management (and easier for beginner to get right). 3) Write struct + operator in C++ (you can use verbatim-C-code (https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#including-verbatim-c-code) and wrap it with Cython - it is easier than try to define them in Cython. – ead Apr 23 '20 at 21:13
  • Thank you for your answer. It's my first post here so I understood that I had to re-edit it after the closure instead of posting a new question. Sorry – Alessandro De Maio Apr 24 '20 at 10:37
  • As for you suggestions: 1) ok for less operator 2) that was only an example, the objects I need to put in the priority queue are bigger than the ones I've used in the example, but I will try to follow your suggestions 3) I've tried to do what you suggest, but now the cythonizing of my pyx stops with the error "priority_queue templated type receives 1 arguments, got 3" so I don't know if it's a limit of the C++ wrapping of std:queue in Cython or not – Alessandro De Maio Apr 24 '20 at 10:41
  • You could update question with your current version, because without seeing the code it is hard to tell what goes wrong. – ead Apr 24 '20 at 10:44
  • Somebody was lazy as they has wrapped priority_queue back then: https://github.com/cython/cython/blob/master/Cython/Includes/libcpp/queue.pxd#L15. So easiest solution would be to override the less operator i.e. to provide non-member `bool operator<(myType *t1, myType *t2)` in C-verbatim code. – ead Apr 24 '20 at 11:15
  • btw. your `struct itemCompare` would not compile in C++ - you have to drop the `self`-parameter. – ead Apr 24 '20 at 11:16
  • @ead Any more suggestion? It would be very appreciated. Thanks – Alessandro De Maio Apr 29 '20 at 20:23
  • 1) my suggestion was to provide `bool operator<(myType *t1, myType *t2) (there are * in type definition i.e. for pointers to myType and not for myType-themselfs as you have done), because you put myType* and not myType into the priority queue 2) see https://stackoverflow.com/a/29011204/5769463 for how to wrap a struct. When you get your code right you can self-answer your answer – ead Apr 29 '20 at 20:44

1 Answers1

1

The answer to my question was using the following pxd file:

cimport cython
from libcpp cimport bool

cdef extern from *:
    """
    typedef struct {
        int key;
        float a;
        float b;       
    } myType;

    bool operator<(const myType t1, const myType t2) {
        return t1.key < t2.key;
    }
    """
    ctypedef struct myType:
        int key
        float a
        float b

Thanks to @ead for his help