2

I am debugging a packet loss issue with my C++ program receiving RTP. After my program ran for quite a long time receiving lots of RTP data, it starts to loose packets.

Tracing with Wireshark shows the missing packets, but my application never receives them. It seems the network stack drops them before delivering them to the application. After restarting my application all goes back to normal.

To produce the error condition, I overload the machine with RTP and at that time packet loss happens and for good reason. But even after I stop the overload and start sending at a moderate rate, packet loss still occurs and I have to restart my application to receive all data again.

Is this an issue with Linux receive buffer handling? What Linux stats could I check to see where those missing packets go?

Alexis Wilke
  • 19,179
  • 10
  • 84
  • 156
Gene Vincent
  • 5,237
  • 9
  • 50
  • 86
  • 3
    when you see them in wireshark, they arrive at the network stack. netstat -s can show quite some statistics from which you can often conclude things such as that the OS buffer ran full because your applicatin did not recv() packets fast enough or similar. – PlasmaHH Dec 18 '12 at 11:40
  • @PlasmaHH: Thanks! Which counters in netstat -s would you expect to increase if my app isn't reading fast enough ? – Gene Vincent Dec 18 '12 at 11:42
  • I don't know how RTP works, and the available stats differ in kernel versions, but if its UDP based then Udp:RcvbufErrors would be a strong indicator for that (you might also want to check if the buffer is too small for your wokrload) – PlasmaHH Dec 18 '12 at 11:45
  • 1
    I suggest `netstat -nptua` -> you can see per-app packet statistics. – hate-engine Dec 18 '12 at 12:52
  • I only see a dash for a application name on my OpenSuse 12.2 (Linux 3.4.11). Is there a way for the application to set its name ? – Gene Vincent Dec 18 '12 at 12:56
  • Are you not getting them in the actual socket `recvfrom` or are you not getting them in your application which is on top of a RTP (or SRTP?) stack of some kind? I ask because the stack may be discarding them due to session state, sequencing, etc. – mark Dec 18 '12 at 13:21

1 Answers1

4

You are not consuming your UDP input fast enough. Here are some usual steps to take to mitigate that:

  • Switch to recvmmsg(2) if your kernel supports it to reduce system call overhead,
  • Pre-allocate all memory used during input processing,
  • Profile your app, find hot spots and optimize,
  • Maybe farm processing out to separate threads, but keep lock scope as small as possible,
  • Increase your socket receive buffer (setsockopt(2)),
Nikolai Fetissov
  • 82,306
  • 11
  • 110
  • 171