Winsock local loop-back latency

Question

I'm using tcp sockets to provide interprocess communication between two apps on Windows XP. I chose tcp sockets for various reasons. I'm seeing an average round-trip time of 2.8 ms. That's much slower than I was expecting. Profiling seems to show that the delay is between one app calling send and the other end's blocking recv returning.

I have too apps, a daemon and a client. They are structured like this pseudo code:

Daemon thread 1 (Listens for new connections):

while (1) {
   SOCKET listener_socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
   bind(listener_socket, (SOCKADDR*)&server_info, sizeof(SOCKADDR));
   listen(listener_socket, 1);
   SOCKET client_socket = accept(listener_socket, NULL, NULL);
   closesocket(listener_socket);
   CreateThread(client_thread);
 }

Daemon client_socket thread (listens for packets from client):

char cmdBuf[256];
int cmdBufAmountData = 0;

while (1)
{   
    char recvBuf[128];
    int bytesTransferred = recv(m_clientSocket, recvBuf, sizeof(recvBuf), 0);

    // Copy received data into our accumulated command buffer (commands 
    // may be split across packet boundaries)
    memcpy(cmdBuf + cmdBufAmountData, recvBuf, bytesTransferred);
    cmdBufAmountData += bytesTransferred;

    // See if there is one or more complete commands in cmdBuf 
    // (commands are separated by '\0')
    while (commandExists(cmdBuf, cmdBufAmountData))
    {
        // do stuff with command
        send(m_clientSocket, outBuf, msgLen, 0);

        // Throw away the command we just processed by shuffling 
        // the contents of the command buffer left
        for (int i = 0; i < cmdBufAmountData - cmdLen; i++)
            cmdBuf[i] = cmdBuf[i + cmdLen];
        cmdBufAmountData -= cmdLen;
    }
}

Client thread 1:

start_timer();
send(foo);
recv(barBuf);
end_timer();       // Timer shows values from 0.7ms to 17ms. Average 2.8ms.

Any ideas why the latency is so bad? I suspected Nagel's algorithm, but littering my code with:

BOOL bOptVal = TRUE;
setsockopt(socket, IPPROTO_TCP, TCP_NODELAY, (char*)&bOptVal, sizeof(BOOL));

Doesn't help. Do I need to do this on both the client and daemon sockets (I am doing)?

I'm on a quad core machine with almost no load, no disk activity etc.

@Jimbo: Well, Len just made me realise that actually my "do stuff" is slow. If I remove the do-stuff, I can measure the tcp round-trip-time as averaging 0.17ms. A 16x speed-up. Unfortunately my daemon does nothing now :-( — Andrew Bainbridge, Oct 13 '10 at 15:29

score 2 · Accepted Answer · answered Oct 12 '10 at 16:45

2

Firstly, in your server, the while loop should be around the Accept rather than the listen... You only need to listen once, so, something more like...

SOCKET listener_socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
bind(listener_socket, (SOCKADDR*)&server_info, sizeof(SOCKADDR));
listen(listener_socket, 1);
while (1) {
   SOCKET client_socket = accept(listener_socket, NULL, NULL);
   closesocket(listener_socket);
   CreateThread(client_thread);
 }

Next, yes, if you want to turn off nagle you need to do it on both the accepted server socket and the connected client socket. You can do it just after you connect/accept. So, if you're only setting nagle on one socket then that may be your issue.

Given that you're using TCP I assume you're reading until you have your complete message and not assuming that one send on one side == one recv on the other. (i.e. I assume your code is abbreviated and doesn't show the normal recv loop).

How many clients? How many threads?

answered Oct 12 '10 at 16:45

Len Holgate

21,282
4
45
92

Thanks for the comments. I didn't realise you could reuse the listener socket. Also, I'll double check my NODELAY at both ends. I've added more detail to the client thread recv de-packetizer. – Andrew Bainbridge Oct 13 '10 at 09:59
There is only one client at this point! So the daemon has two threads, and client has several too, but only one communicating with the daemon at once (protected by a mutex). I plan to cope with up to about 3. Scary stuff. – Andrew Bainbridge Oct 13 '10 at 10:02
1

Do you have a timer in your server which shows how much time it takes for the server code to generate the response? – Len Holgate Oct 13 '10 at 11:03
Yes. Unfortunately it was wrong, due to lack of precision. And I suspected my network code because that was the stuff I was unsure about. Thanks for the kick in the right direction. – Andrew Bainbridge Oct 13 '10 at 12:55
BTW, I suspect the closesocket in your example should be after the while loop. – Andrew Bainbridge Oct 13 '10 at 12:55
yeah, you're right, that code of mine there has never seen a compiler, let alone a unit test ;) – Len Holgate Oct 13 '10 at 13:23

score 1 · Answer 2 · answered Oct 13 '10 at 00:43

1

And you shouldn't close the listening socket until you want to exit your server.

I would have a look at named pipes rather than sockets if you don't mind being wedded to the Windows API.

answered Oct 13 '10 at 00:43

user207421

305,947
44
307
483

Winsock local loop-back latency

2 Answers2