2

I'm trying to send image numpy arrays using the ZMQ pub/sub model. I followed the documentation here:

However, I realized that there would be a buildup of frames in my subscriber as the subscriber wouldn't be able to process the frames at the rate at which the publisher is sending due to the operations done on the subscriber side for the frames.

Since the documentation uses a multipart message to send the image array, I am unable to use the CONFLATE option on set_sockopt as that makes it only keep a part of the multipart.

My question: Is there a way for the subscriber to keep only the latest multipart message?

Benyamin Jafari
  • 27,880
  • 26
  • 135
  • 150
ksuhartono97
  • 35
  • 7

2 Answers2

1

In the end I sort of solved this by repeatedly subscribing and unsubscribing from the ZMQ socket. 

# This is run every time the subscriber receive function is called
socket.setsockopt(zmq.SUBSCRIBE, '')
md = socket.recv_json()
msg = socket.recv()
socket.setsockopt(zmq.UNSUBSCRIBE, '')

Essentially, I made it so that my subscriber socket doesn't care about the other messages that come in other than the one message that it has received until the next time it tries to grab a message.

I don't believe that this is the best solution for this problem, as there are costs involved when repeatedly subscribing and unsubscribing. Hoping that there might be a better way to do this but so far I haven't been able to find it.

ksuhartono97
  • 35
  • 7
-1

Are you looking for zmq.CONFLATE option ("Last Message Only")?

Something like this in subscriber side:

context = zmq.Context()
socket = context.socket(zmq.SUB)

socket.setsockopt(zmq.SUBSCRIBE, '')
socket.setsockopt(zmq.CONFLATE, 1)  # last msg only.
socket.connect("tcp://localhost:%s" % port)  # must be placed after above options.

Relevant post

Learn more

Benyamin Jafari
  • 27,880
  • 26
  • 135
  • 150
  • 1
    something like `zmq.CONFLATE` but `zmq.CONFLATE` doesn't support multi-part messages, so it doesn't work for my case. – ksuhartono97 Sep 15 '19 at 12:27
  • @ksuhartono97 Apparently that's right. There are two other option: `setsockopt(zmq.SNDHWM, 1)` in publisher side, and `setsockopt(zmq.RCVHWM, 1)` in subscriber side. – Benyamin Jafari Sep 15 '19 at 12:43
  • Tried that as well but the sockets seem to ignore that setting, it was still sending more than the single multipart message that I want to keep. – ksuhartono97 Sep 17 '19 at 02:17