1

I'd like to use python to build an app that is similar to zeroMQ's ventilator / sink scheme

Suppose that we have 10 Workers, all running on the same multi-core server.

Let's say that every 2[sec] or so, each Worker pushes to the Sink a message of size 5[MB]. So, the Sink has to handle a total of 50[MB] ( = 10 x 5[MB] ) every 2[sec] or so.

Had the 10 Workers been on different machines, I know that the network could have been a potential bottleneck.

Had the 10 Workers had to write their data to disk (I/O), I know that the disk could have been a potential bottleneck.

Given the fact that all the 10 Workers are on the same machine, what bottlenecks should one expect?

For example, can the same 10 Workers each push a message of size 10[MB] every 2[sec] or so? Can they push a message of size 20[MB] every 2[sec] or so?

What are zmq's limitations?

What types of bottlenecks should one expect when using python and zeroMQ in a Linux environment?

Wooble
  • 87,717
  • 12
  • 108
  • 131
user3262424
  • 7,223
  • 16
  • 54
  • 84

1 Answers1

2

Using PUSH/PULL on the same server I've been been able to max out writing to a raid array @ 400MB/sec (bottle-necked by write speed). There are 10GbE benchmark results here. I'd suggest constructing some simple benchmarks, performance is going to be dependent on a lot of factors like message format, size, etc.

For example a completely trivial benchmark shows zeromq capable of sending 100 10mb messages in 12.3 ms on my machine:

# server
import zmq

context = zmq.Context()
reciever = context.socket(zmq.PULL)
reciever.bind('tcp://127.0.0.1:5555')

while True:
    reciever.recv()

# client
import os, zmq

context = zmq.Context()
pusher = context.socket(zmq.PUSH)
pusher.connect('tcp://127.0.0.1:5555')

message = ' ' * 10485760

>>> %timeit pusher.send(message)
100 loops, best of 3: 12.3 ms per loop
Zach Kelling
  • 52,505
  • 13
  • 109
  • 108
  • zeekay, thank you. Please note -- my example does not involve the **Workers** writing to disk -- simply sending the data over to the **Sink** (meaning, no disk is involved). Any hints what types of bottlenecks I should expect? – user3262424 Jul 15 '11 at 01:58
  • It's hard to say, you could bottleneck yourself in a lot of ways unrelated to zeromq. What are your workers doing? How are you constructing messages? What protocol are you using? I'd construct a simple benchmark and test performance in your particular case. In my experience you can transfer hundreds of megabytes/sec from a single worker, assuming the messages can be constructed/processed fast enough. I have no idea what the upper limit of zeromq's performance is. – Zach Kelling Jul 15 '11 at 02:07
  • Updated answer with a simple benchmark demonstrating zeromq's performance sending 10Mb messages. – Zach Kelling Jul 15 '11 at 02:30
  • zeekay, thank you. I tried to play with `zeroMQ`, sending over messages of size 4[MB] or so, using code similar to yours. It looks like **some messages ARE NOT RECEIVED** at the server; They are sent, but I do not see them RECEIVED. why is that? – user3262424 Jul 15 '11 at 02:37
  • You might want to ask another question about lost messages using `PUSH`/`PULL` (or whatever you used). – Zach Kelling Jul 15 '11 at 02:39
  • Yes -- here it is: http://stackoverflow.com/questions/6702187/python-zeromq-push-pull-lost-messages – user3262424 Jul 15 '11 at 02:48
  • If you are concerned with memory performance, you might want to be using `copy=False`, which allows you to send/recv messages without any in-memory copies. Using the identical code above: `In [7]: %timeit pusher.send(message)` `10 loops, best of 3: 20.5 ms per loop` `In [8]: %timeit pusher.send(message, copy=False)` `100000 loops, best of 3: 9.37 us per loop` So non-copying sends return *much* faster for large messages. Note that send returning does *not* mean that the message has actually left over the wire. The actual network traffic is always hidden by 0MQ. – minrk Nov 10 '11 at 03:15
  • I don't get how to use `%timeit`, or the `In [7]:` etc. syntax. Can you please clarify how to use these? – Wim Leers Mar 21 '12 at 19:09