Zeromq memory leak (pyzmq)

Question

Hi I'm trying to send large packets with ZeroMQ using the ventilator/worker/sink pattern.

I try adding workers. Each time, the sink process memory usage increases a little. Then it reaches a tipping point at about 6 or 7 workers where suddenly memory increases exponentially until it dies with:

> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug Assertion failed: (msg_->flags | ZMQ_MSG_MASK) == 0xff (zmq.cpp:211)
> Python(42410,0xaccb8a28) malloc: *** mmap(size=3559424) failed (error
> code=12)

Here is the code (showing only the worker/sink pattern):

import sys
import resource
import zmq
import time

context = zmq.Context()


if sys.argv[1] == 'worker':
    # Socket to send messages to

    sender = context.socket(zmq.PUSH)
    sender.connect("tcp://localhost:5558")

    while True:
        msg = 'x' * 3559333
        time.sleep(.01)
        sender.send(msg)
else:
    # Socket to receive messages on

    receiver = context.socket(zmq.PULL)
    receiver.bind("tcp://*:5558")
    while True:
        msg = receiver.recv()

        print msg[0:5], len(msg), resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

Is this simply a lack of hardware resources? A backlog of data? Or is there a way of avoiding this?

I'm running OSX Mountain Lion with 16gb memory and Python 2.7 with zmq 2.2.0.1.

Thanks

Answer 1

Is this simply a lack of hardware resources?

Well, let's do the math. Each worker is sending 3.3MB every 10ms. Or about 300mb a second. Now you add more workers. By the time you are up to 5 workers you are sending about 1.5GB per second.

I think you have found the performance limit for your machine. When the sink process is running on the same machine as all the workers, it is capable of consuming somewhere between 1-2GB per second. When the data is coming in faster than that the queue's build up in the sink process faster than they can be emptied and you run out of memory.

Or is there a way of avoiding this?

Send smaller messages? Less frequently? :) Or put the workers and sink process on different machines. Remember the workers are stealing CPU resources from the sink. If this is a quad core machine, then with the sink plus up to 3 workers, the OS is probably allocating almost all of a processor core to each process.

Once you add the 4th, 5th, 6th worker, the OS cannot give 100% of a core to any process. They have to start sharing, so the sink slows down even as the rate of messages speeds up. That would explain the tipping point you are seeing where memory usage increases exponentially.

Hmmm - which suggests an interesting experiment. Can you configure your mac so that the sink process runs at a really high priority? That might give better results. I have never tried this myself, but see the following link for ideas ... https://discussions.apple.com/thread/1491812?start=0&tstart=0