Python ZeroMQ PUSH/PULL -- Lost Messages?

Question

I am trying to use python with zeroMQ in PUSH / PULL mode, sending messages of size 4[MB] every few seconds.

For some reason, while it looks like all the messages are sent, ONLY SOME of them appear to have been received by the server. What am I missing here?

Here's the code for the client -- client.py

import zmq
import struct

# define a string of size 4[MB] 
msgToSend = struct.pack('i', 45) * 1000 * 1000 

context = zmq.Context()
socket = context.socket(zmq.PUSH)
socket.connect("tcp://127.0.0.1:5000")

# print the message size in bytes
print len(msgToSend)

socket.send(msgToSend)

print "Sent message"

And here is the code for the server -- server.py

import zmq
import struct

context = zmq.Context()
socket = context.socket(zmq.PULL)
socket.bind("tcp://127.0.0.1:5000")

while True:
    # receive the message
    msg = socket.recv()

    print "Message Size is: {0} [MB]".format( len(msg) / (1000 * 1000) )

What am I missing? How do I guarantee that messages are always sent and not lost?

In case it matters, I am using Ubuntu 10.04 32bit, Core Duo machine with 2[GB] RAM.

NOTE: I tried the same example using RabbitMQ and everything works well -- no message is lost. I am perplexed as I often hear praises of zeroMQ. Why did it fail where RabbitMQ succeed?

score 16 · Accepted Answer · edited Feb 26 '14 at 17:28

16

The problem is that when the program exits, the socket gets closed immediately and garbage collected with an effective LINGER of 0 (i.e. it throws any unsent messages away). This is a problem for larger messages because they take longer to send than it takes for the socket to be garbage collected.

You can avoid this by putting a sleep(0.1) just before the program exits (to delay the socket and context being garbage collected).

socket.setsockopt(zmq.LINGER, -1) (which is the default) should avoid this problem, but it doesn't for some reason that I haven't had time to investigate.

edited Feb 26 '14 at 17:28

Community

1
1

answered Jul 15 '11 at 07:59

cwb

925
5
6

cwb, thank you. It works, but it appears that `rabbitMQ` is so much more relaible. I can send even 40-60 [MB] of data fluently, while in `zeroMQ` things get stuck / do not work as reliably. – user3262424 Jul 15 '11 at 15:02
1

Fair enough. I don't know whether to blame 0MQ or pyzmq in this case; the end result is trouble for your regardless. Whether the 0MQ benefits over RabbitMQ are worth it is your decision of course. 40-60MB of data shouldn't be a problem for 0MQ either; but something here is tripping it up... – cwb Jul 15 '11 at 15:22
1

I know `0MQ` should work (and perhaps, works even faster than `rabbitMQ`) -- yet, from my limited testing, unless one needs **extreme speed** -- `rabbitMQ` is much more reliable. – user3262424 Jul 15 '11 at 15:48
This python issue seems to apply to C++ as well. I had the same problem with 2 threads writing to the same destination (tcp://host:port), and each thread is creating and destroying the socket quickly. It appears that when one thread destroys its socket, it has a side effect of destroying the unsent messages from the other socket (even if the "sockets" are thread local). – Mark Lakata May 29 '14 at 00:39

score 1 · Answer 2 · answered Jul 15 '11 at 04:42

It's conceivable that you are running out of memory (depending how you are sending messages, whether they are being consumed fast enough, etc). You can use socket.setsockopt(zmq.HWM) to set HWM to a sane value and prevent zeromq from storing too many messages in the outgoing buffer. With this in mind, consider slightly modified examples:

# server
...
counter = 0
while True:
    ...receive the message
    counter += 1
    print "Total messages recieved: {0}".format(counter)

# client
socket.setsockopt(zmq.HWM, 8)
for i in range(1000):
    socket.send(msgToSend)

And then running 10 test clients:

for i in {1..10}; do
    python client.py &
done

From the server you can see all the messages are recieved:

Total messages recieved: 9998
Total messages recieved: 9999
Total messages recieved: 10000

zeekay, thank you. It appears that rabbitMQ is so much more relaible that `zeroMQ`. I just succeeded (although it took some time) to send 320 [MB]. `zeroMQ` get stuck / does not work as reliably. — user3262424, Jul 15 '11 at 15:04

Python ZeroMQ PUSH/PULL -- Lost Messages?

2 Answers2

Linked