4

On Redhat Linux, I have a multicast listener listening to a very busy multicast data source. It runs perfectly by itself, no packet losses. However, once I start the second instance of the same application with the exactly same settings (same src/dst IP address, sock buffer size, user buffer size, etc.) I started to see very frequent packet losses from both instances. And they lost exact the same packets. If I stop the one of the instances, the remaining one returns to normal without any packet loss.

Initially, I though it is the CPU/kernel load issue, maybe it could not get the packets out of buffer quickly enough. So I did another test. I still keep one instance of the application running. But then started a totally different multicast listener on the same computer but use the second NIC card and listen to a different but even busier multicast source. Both applications run fine without any packet loss.

So it looks like one NIC card is not powerful enough to support two multicast applications, even though they listen to exact the same thing. The possible cause to the packet loss problem might be that, in this scenario, the NIC card driver needs to copy the incoming data to two sock buffers, and this extra copy task is too much for the ether card to handle so it drops packets. Any deeper analysis on this issue and any possible solutions?

Thanks

welch
  • 61
  • 1
  • 2
  • It is really weird. The NIC card won't copy from NIC device driver to sockets immediately. When NIC receives a multicast frame, the multicast service routine will do filtering and forward the packet to protocol stack routines. The protocol stack routines is responsible to copy the UDP packet to each socket buffer. What will happen if you let the two instances join the same multicast address but on different interfaces? – badawi Jul 08 '11 at 18:57
  • I could not do this test because only this NIC port is connected to the data source. But the test I did (mentioned in the question) suggested that it should work. But I really want to make it works on the same NIC card. – welch Jul 08 '11 at 19:54

1 Answers1

1

You are basically finding out that the kernel is inefficient at fan-out of multicast packets. Worst case scenario the code is for every incoming packet allocating two new buffers, the SKB object and packet payload, and copying the NIC buffer twice.

Pick the best case scenario, for every incoming packet a new SKB is allocated but the packet payload is shared between the two sockets with reference counting. Now imagine what happens when two applications, each on their own core and on separate sockets. Every reference to the packet payload is going to cause the memory bus to stall whilst both core caches have to flush and reload, and above that each application is having to kernel context switch back and forth to pass the socket payload. The result is terrible performance.

You aren't the first to encounter such a problem and many vendors have created solutions to it. The basic design is to limit the incoming data to one thread on one core on one socket, then have that thread distribute the data to all other interested threads, preferably using user space code building upon shared memory and lockless data structures.

Examples are TIBCO's Rendezvous and 29 West's Ultra Messaging showing a 660ns IPC bus:

http://www.globenewswire.com/newsroom/news.html?d=194703

Steve-o
  • 12,678
  • 2
  • 41
  • 60
  • Hi Steve, Any open source solution available to address this problem? – bdubey Jun 09 '15 at 07:11
  • @bdubey check out [aeron](http://highscalability.com/blog/2014/11/17/aeron-do-we-really-need-another-messaging-system.html) or [disruptor](https://lmax-exchange.github.io/disruptor/). – Steve-o Jun 14 '15 at 23:44