SocketCAN with C++ on Raspberry Pi: messages lost when read is delayed

Question

In a C++ application running on a Raspberry Pi, I am using a loop in a thread to continuously wait for SocketCAN messages and process them. The messages come in at around 1kHz, as verified using candump.

After waiting for poll() to return and reading the data, I read the timestamp using ioctl() with SIOCGSTAMP. I then compare the timestamp with the previous one, and this is where it gets weird: Most of the time, the difference is around 1ms, which is expected. But sometimes (probably when the data processing takes longer than usual or gets interrupted by the scheduler) it is much bigger, up to a few hundred milliseconds. In those instances, the messages that should have come in in the meantime (visible in candump) are lost.

How is that possible? If there is a delay somewhere, the incoming messages get buffered? Why do they get lost?

This is the slightly simplified code:

while(!done)
{
    struct pollfd fd = {.fd = canSocket, .events = POLLIN};
    int pollRet = poll(&fd, 1, 20); // 20ms timeout
    if(pollRet < 0)
    {
        std::cerr << "Error polling canSocket" << errno << std::endl;
        done = true;
        return;
    }
    if(pollRet == 0) // timeout, never happens as expected
    {
        std::cout << "canSocket poll timeout" << std::endl;
        if(done) break;
        continue;
    }

    struct canfd_frame frame;
    int size = sizeof(frame);
    int readLength = read(canSocket, &frame, size);
    if(readLength < 0) throw std::runtime_error("CAN read failed");
    else if(readLength < size) throw std::runtime_error("CAN read incomplete");

    struct timeval timestamp;
    ioctl(canSocket, SIOCGSTAMP, &timestamp);
    uint64_t timestamp_us = (uint64_t)timestamp.tv_sec * 1e6 + (uint64_t)timestamp.tv_usec;
    
    static uint64_t timestamp_us_last = 0;
    if((timestamp_us - timestamp_us_last) > 20000)
    {
        std::cout << "timestamp difference large: " << (timestamp_us - timestamp_us_last) << std::endl; // this sometime happens, why?
    }
    timestamp_us_last = timestamp_us;
    
    // data processing
}