1

I am facing a strange issue with ZMQ, which I'm just not able to debug. These are the components:

  • Java ZMQ Server - Almost an exact copy of this example. There are a hundred worker threads.

  • PHP Client - Simple request reply with a REQ socket. This is the request flow:

    $zcontext = new ZMQContext();
$socket = new ZMQSocket($zcontext, ZMQ::SOCKET_REQ);
$socket->connect(<address>);
$startTime = microtime(true);
$socket->send(<request>);
$result = $socket->recv();
$totalTime = microtime(true) - $startTime;

The ZMQ sockets use TCP and both the server and client are on the same machine.

The PHP script is served by apache and I am load testing using apache benchmark. I make 5000 requests with a concurrency of 200. On the PHP client I log the time it takes for the request reply ($totalTime). In most of the cases, this time is very low (sub 500ms), but occasionally it takes a really long time - sometimes even 60 secs (for send + receive).

I added some extra logging to find out where the issue is happening, and it turns out that whenever it takes really long, almost all the time is between PHP's send and Java's receive - so packets are taking really long to reach the server.

I'm not setting any special ZMQ settings, or otherwise doing anything unusual so I don't know what is causing the issue. It should also be noted that the issue persists even at lower concurrencies (I tested at 100 and 150 too), but the max request times are lower.

Sorry if the question seems vague - I'll provide any other details that may be needed.

Jayanth Koushik
  • 9,476
  • 1
  • 44
  • 52
  • 1
    In your extra logging, have you definitely confirmed that the lag is not coming in PHP/Apache spinning up the script at high load, prior to sending off the message to the server? – Jason Apr 16 '15 at 18:27
  • No, and I'm not sure how I would be able to do that. Once I call `socket->send` from PHP, I cannot "track" the message until I receive it on the server right? I just have the time difference between `socket->recv` and `socket->send` (which is really high sometimes). – Jayanth Koushik Apr 17 '15 at 03:13
  • OK, I understand a little better. You *can* track a message, [see some details here](http://zeromq.org/whitepapers:traffic-monitoring-v20). The goal is to see which half of the system is lagging, which will help guide further exploration. – Jason Apr 17 '15 at 14:11
  • Oh cool. Will try that on Monday - see what comes up. – Jayanth Koushik Apr 17 '15 at 19:09
  • Some updates - For a single PHP process, the problem occurs only with the very first request. Furthermore, the problem goes away if I add a 1 sec sleep after socket connect. Could it perhaps be a memory issue? – Jayanth Koushik Apr 20 '15 at 14:44

0 Answers0