Problem
I am facing what I would assume is a common problem where I have a publisher which publishes on topics that are both high and low-volume (i.e. both topics where dropping is okay when subscriber/network is slow because they are high-frequency and topics where updates are slow and I never want to drop messages).
Possible Solutions
Unbound queue on Publisher side
This might work, but the high-volume topics may be so fast that they flood memory. It is desirable for high-volume topics to be dropped and low-volume topics to be protected.
One PUB socket per high-volume topic or all high-volume topics or for every topic
Based on my understanding of ZeroMQ's queueing model, with a queue for each connection both on the publisher side and on the subscriber side, the only way to protect my low-volume topics from being dropped and pushed out by the high-volume topics right now is to create a separate PUB socket for each or all high-volume topics and somehow communicate that to subscribers who will need to connect to multiple endpoints from the same node.
This complicates things on the subscriber side, as they now need prior knowledge of mappings between ports and topics or they need to be able to request an index from the publisher node. This then requires that the port numbers are fixed or that every time a subscriber has a connection issue, it checks the index to see if the port changed.
Publisher- and Subscriber- Side Topic Queues Per Connection
The only solution I can see, at the moment, is to create a queue for each subscriber & topic on the publisher side and on the subscriber side, hidden away inside a library so neither side needs to think about it. When messages for a specific topic overflow, they can still be dropped without pushing out messages for other topics. A separate ordered dictionary would need to be used to maintain queue pull/get order by a worker feeding messages into the PUB socket or pulling events out from the subscriber-side.
This solution only works if I know when the ZeroMQ queue on the publisher side is in a mute state and will drop messages, so I know to hold off "publishing" the next message which will probably be lost. This can be done with the option, ZMQ_XPUB_NODROP (http://api.zeromq.org/master:zmq-setsockopt).
This will probably work, but it is non-trivial, probably slower than it could be because of the language I use (Python), and the kind of thing I kind of would have expected a messaging library to handle for me.
