How to measure precise or approximate latency of receiving packet

Question

I am trying to measure latency when a packet comes in Rx buffer and is copied to application memory. I am measuring it with this code:

struct timespec start, end;

clock_gettime(CLOCK_REALTIME, &start);
recvfrom(sock, msg, msg_len, 0, &client, &client_addrlen);
clock_gettime(CLOCK_REALTIME, &end);

I know this cannot precisely measure the latency. However, I can calculate average latency by receiving many packets, measuring each one, and calculating them. Is there any method to measure latency more precisely? (e.g., latency = (time when recvfrom() is done) - (time when NIC receives a packet from))

For a device and device driver, I am using Mellanox connectx-3 and mlx4_en.

Answer 1

I was able to get an almost precise number with recvmsg().

Reference

• The most helpful link (https://stackoverflow.com/a/47329376/5215330)
• Network timestamping (https://www.kernel.org/doc/Documentation/networking/timestamping.txt)
• Linux Precision Time Protocol (http://linuxptp.sourceforge.net/)

Code

I am reproducing the code from the first link. This code is not a ready-to-run, but just a snippet from the working code.

static struct timespec handle_time(struct msghdr *msg) {
    struct cmsghdr *cmsg = CMSG_FIRSTHDR(msg);
    struct scm_timestamping *ts = (struct scm_timestamping *)CMSG_DATA(cmsg);
    return ts->ts[0];
}

...
char ctrl[64];
char *msg = malloc(64);

int val = SOF_TIMESTAMPING_RX_HARDWARE | SOF_TIMESTAMPING_RX_SOFTWARE
        | SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_RAW_HARDWARE;

setsockopt(sock_fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));

// user buffer
struct iovec iov = {
    .iov_base = msg,
    .iov_len = msg_len,
};

// ancillary message header 
struct msghdr m = {
    .msg_name = &client_addr,           // struct sockaddr_in
    .msg_namelen = client_addrlen,      // socklen_t
    .msg_iov = &iov,
    .msg_iovlen = 1,
    .msg_control = &ctrl,
    .msg_controllen = sizeof(ctrl),
};

while (1) {
    memset(msg, 0, msg_len);
    num_received = recvmsg(sock_fd, &m, 0);
    start = handle_time(&m);
    clock_gettime(CLOCK_REALTIME, &end);

    if (verbose) {
        double elapsed_time = time_diff(start, end) / 1000;
        total_elapsed += elapsed_time;
        count++;
        printf("%f us %f us\n", elapsed_time, total_elapsed / count);
    }

    if (sendto(sock_fd, msg, msg_len, 0, (struct sockaddr *) &client_addr, client_addrlen) < 0) {
        perror("\nMessage Send Failed\n");
        fprintf(stderr, "Value of errno: %d\n", errno);
    }
}

The key point is to use setsockopt() and recvmsg(). The key mechanism is when you set an option for a certain socket FD, the kernel will set a timestamp based on timestamp flag. After you set them, if you receive a message with struct msghdr, the kernel will audit the timestamp in a way of SW or HW. When you look into the data, you would be able to get 3 timestamps. These information can be explained below:

The structure can return up to three timestamps. This is a legacy feature. At least one field is non-zero at any time. Most timestamps are passed in ts[0]. Hardware timestamps are passed in ts[2]. ts[1] used to hold hardware timestamps converted to system time. Instead, expose the hardware clock device on the NIC directly as a HW PTP clock source, to allow time conversion in userspace and optionally synchronize system time with a userspace PTP stack such as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst.

See 2.1 from Documentation/networking/timestamping.txt for detail.

If you want to see HW timestamp then you need to have a specific HW (refer this comment) and turn its feature with ioctl(). However, there is a convenient tool called linuxptp, which does this job.