latency of recvfrom varies with the frequency it is called

Question

I don't understand the behavior of recvfrom in the following situation: I have a loop that sends and receives udp packets and sleeps a bit; depending on that bit receiving (recvfrom) latency varies. If I sleep for 22 milliseconds or more the latency is around 170 microsecond, if I sleep 21 milliseconds or less the latency is around 1 millisecond. What am I missing? Why the latency varies so much? I thought about interrupt coalescence but I checked with ethtool and it is disabled.

The code is from https://gist.github.com/suyash/0f100b1518334fcf650bbefd54556df9 and slightly modified.

client.c:

#include <arpa/inet.h> 
#include <stdio.h>
#include <string.h>
#include <sys/socket.h>
#include <unistd.h>
#include <errno.h>
#include <time.h>
#include <inttypes.h>

#define TIMESPEC_NSEC(ts) ((ts)->tv_sec * 1000000000ULL + (ts)->tv_nsec)

uint64_t realtime_now()
{
   struct timespec now_ts;
   clock_gettime(CLOCK_MONOTONIC, &now_ts);
   return TIMESPEC_NSEC(&now_ts);
}

int main() {
   const char* server_name = "192.168.1.240";
   const int server_port = 8877;

   struct sockaddr_in server_address;
   memset(&server_address, 0, sizeof(server_address));
   server_address.sin_family = AF_INET;
   inet_pton(AF_INET, server_name, &server_address.sin_addr);
   server_address.sin_port = htons(server_port);
   int sock;
   if ((sock = socket(PF_INET, SOCK_DGRAM, 0)) < 0) {
       printf("could not create socket\n");
       return 1;
   }

  const char* data_to_send = "...";
  struct timespec ts;
  ts.tv_nsec = 21 *1000*1000;
  ts.tv_sec =  0;
  uint64_t t0,t1;
  while(1)
  {
    int len =sendto(sock, data_to_send, strlen(data_to_send), 0,
                    (struct sockaddr*)&server_address,sizeof(server_address));

    char buffer[100];
    t0 = realtime_now();
    recvfrom(sock, buffer, len, 0, NULL, NULL);
    t1 = realtime_now();
    printf("%" PRIu64 "\n",t1 - t0);
    buffer[len] = '\0';
    printf("recieved: '%s'\n", buffer);
    int err = nanosleep(&ts, NULL);
    if (err == EINTR)
    {
        err = 0;
    }
    if(err!=0)
    {
        perror("nanosleep failed");
        break;
    }
  }
  close(sock);
  return 0;
 }

server.c

#include <arpa/inet.h>
#include <netinet/in.h>
#include <stdbool.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char *argv[]) {
    int SERVER_PORT = 8877;
    struct sockaddr_in server_address;
    memset(&server_address, 0, sizeof(server_address));
    server_address.sin_family = AF_INET;
    server_address.sin_port = htons(SERVER_PORT);
    server_address.sin_addr.s_addr = htonl(INADDR_ANY);

    int sock;
    if ((sock = socket(PF_INET, SOCK_DGRAM, 0)) < 0) {
        printf("could not create socket\n");
        return 1;
    }

    if ((bind(sock, (struct sockaddr *)&server_address,
              sizeof(server_address))) < 0) {
        printf("could not bind socket\n");
        return 1;
    }
    struct sockaddr_in client_address;
    socklen_t client_address_len = sizeof(client_address);
    while (true) {
        char buffer[100];
        int len = recvfrom(sock, buffer, sizeof(buffer), 0,
                           (struct sockaddr *)&client_address,
                           &client_address_len);

        buffer[len] = '\0';
        sendto(sock, buffer, len, 0, (struct sockaddr *)&client_address,
               sizeof(client_address));
    }

    return 0;
}

Have you seen _[this post](https://stackoverflow.com/questions/47618619/periodic-latency-spikes-from-udp-socket-caused-by-periodic-sendto-recvfrom-d)_? — ryyker, Jul 24 '18 at 14:38
I didn't, thanks. It seems a different case. I have no threads and it is not a matter of spikes, latency just gets worse if client loop runs more frequently. — ieio, Jul 24 '18 at 14:48
Have you used proper scheduler for your processes ? increased their priority ? pinned your processes to a dedicated CPU core ? — Yann Droneaud, Jul 24 '18 at 17:11
I set affinity for ethernet irq, the system is not loaded at all. — ieio, Jul 25 '18 at 07:33
Why sleep at all? What is to be gained? You're in blocking mode: `recv()` will return when there is data available to be read, and not before. Sleeping is just literally a waste of time. — user207421, Jul 25 '18 at 10:00
I don't think so, I want to wait a while before sending a new packet, if I do not sleep I will send send a packet just after having received it flooding the server. — ieio, Jul 25 '18 at 10:10

Maxim Egorushkin · Answer 1 · 2018-07-25T09:14:07.797

0

Note that you send packets to 192.168.1.240 through the local network over the wire (unless you set up a host-local virtual network). The packets first go through your switch/router and then to your server. And the reverse way back. So, you effectively measure a round-trip time here, not the latency of recvfrom.

edited Jul 25 '18 at 09:14

answered Jul 24 '18 at 16:41

Maxim Egorushkin

131,725
17
180
271

right the latency of the execution of recvfrom, in this case is round-trip time. Between the client and the server there is just a Netgear 10GB switch, no router. – ieio Jul 25 '18 at 07:31

latency of recvfrom varies with the frequency it is called

1 Answers1