3

I am currently doing a latency benchmark of some communication middlewares and I found that ZeroMQ REQ and REP sockets are surprisingly slow. For example I was expecting that ZeroMQ would be faster than ZeroC Ice. For completeness here are the results of the tests so far:

  1. Ice 7600 messages per second
  2. ZeroMQ 4300 - 4500 messages per second

The message is a remote invocation with a data structure as a parameter. The data structure contains fields of basic types (int, float, double, boolean, string). At first I suspected that serializing the data with Protocol Buffers could be a bottleneck for ZeroMQ, but then I tested using empty messages and the results are fairly similar. Ice is faster even if ZeroMQ is sending empty messages.

Nevertheless I want to give ZeroMQ a fair chance. In that spirit I want to know if I can improve the speed of REQ and REP in any way. Maybe I should use other sockets? The only restriction is that the communication should be RPC-like. The client shouldn't do any work while it expects the result of its message.

Here is the code of my implementation. I provide the Java version of the benchmark, but the results are fairly similar for Python(a bit slower) and C++ (a bit faster):

Client

...
MQ.Context context = ZMQ.context(1);
ZMQ.Socket socket = context.socket(ZMQ.REQ);
socket.connect("tcp://*:5555");

for(int i=0; i<measuredIterations; i++){
    t0 = System.nanoTime();
    socket.send("",0);
    socket.recv(0);
    rtt = System.nanoTime() - t0;
    ...
}

Server

ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket socket = context.socket(ZMQ.REP);
socket.bind("tcp://*:5555");


while (!Thread.currentThread().isInterrupted()){
    socket.recv(0);
    socket.send("",0);
}

The rest of the code is basically for calculating the average latency and the messages per second.

Maybe ZeroMQ is just slow for this communication pattern and it excels in many-to-many communication patterns...

Note: I "warmed" both middlewares before recording the results. Nevertheless ZeroMQ is always slower than Ice.

EDIT1: I increased the number of warming iterations to 10000. The new results are:

  1. Ice 8100 messages per second (converges to 8250 with more warming)
  2. ZeroMQ + Protocol Buffers converges to 4650 messsages per second
Renato Sanhueza
  • 534
  • 7
  • 27
  • 2
    "ZeroMQ surprisingly slow" - you are rather far from being a first user to make this observation. :-) – oakad Jan 02 '18 at 00:42
  • Is this related to some MCVE-case based A/B-test comparison, to have some relevance on comparing apples to indeed apples? Would welcome both sides of the A/B-test to have a common fashion of a benchmark -- sending rather a 10k block of messages, not a single-shot handling, contain all of transport-classes, covering each of `[ inproc://, ipc://, tcp://, pgm://, epgm://, vmci:// ]` one after another, with a scaling, covering a growth path from about 2 - 5 recipients to some 200 - 500. That would mean a scientifically fair step for a fact based repeatable experiment comparison. No value otherwise – user3666197 Jan 02 '18 at 09:11
  • @user3666197 I agree that comparing a RPC middleware with ZeroMQ is not an easy task. In this case I tried to simulate a RPC with ZeroMQ which is one of the use case required. Maybe ZeroMQ would perform better in other use cases like publish-subscribe of events... – Renato Sanhueza Jan 02 '18 at 15:49
  • ... I selected tcp transport because it was the most fitting for the use case, but I will explore the rest and check if anything changes. The test warmed the middleware during 100 messages and then it sent 100000 messages in a synchronous way. I agree that a full experiment would consider the scaling factor, but this experiment is intended to be only a simple latency benchmark for now. – Renato Sanhueza Jan 02 '18 at 15:49
  • Might be a fair step to also **disconnect and re-connect the LAN-cable** over the network hop ~ 3-5 times during the run of the experiment, so as to indeed compare the apples to apples. Good luck with the testing +1. – user3666197 Jan 02 '18 at 16:41
  • Interesting. Maybe ZeroMQ is slower in this case because of the Performance/Reliability trade-off. – Renato Sanhueza Jan 02 '18 at 17:11

0 Answers0