1

How to make a TCP dump where it is guaranteed that all the packets that really pass through the network are captured, and nothing is missed?

Details: We have an issue with 3rd party vendor who provides a solution on top of SCTP stack, which he also implements.
Under quite high throughput (52 000 messages/sec, average message size is 500 bytes) the SCTP link breaks.
We believe that the bug is in the vendor SCTP stack.
But the vendor says, this happens because SCTP stack sends a message, doesn't receive ACK on it, sends a number of retransmits, doesn't receive ACKs on them as well and closes the SCTP link.
So the vendor says, this is the network which is guilty, because it loses packets.

In the TCP dumps on both sides, client and server we see that the original messages reaches the server and see that the server doesn't answer with ACK. But the vendor says that TCP dumps are not reliable, that when capturing a TCP dump, some packets could be not captured, because libpcap library works only within one hardware thread, its power can be not enough to log all the packets.

Technical data: 52 000 messages/sec, average message size is 500 bytes, so 26 MB/sec in total, 4 SCTP links are used.
Hardware: CPU E5-2670, 2.6 GHz, 8 HW threads
Network: 10 GBit, the traffic is between HP blades, which are located in one rack.
RHEL 6.

Neighbour
  • 97
  • 10
  • 2
    Are you actually losing packets in the dump? I think tcpdump will tell you if any packets were not captured. – Barmar Jul 03 '14 at 10:20
  • @Barmar Actually there are not captured packets: /usr/sbin/tcpdump -w 1.pcap tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes ^C392812 packets captured 395167 packets received by filter 2332 packets dropped by kernel – Neighbour Jul 03 '14 at 10:36
  • 1
    Check this thread: http://www.linuxquestions.org/questions/linux-networking-3/tcpdump-and-dropped-packets-72369/ You need to run the capture on hardware which is capable of capturing all packets. – hek2mgl Jul 03 '14 at 10:42
  • @hek2mgl We already don't resolve DNS names and ports. What can I do with existing hardware, I gave the spec above? – Neighbour Jul 03 '14 at 10:53
  • What if you just use a slower network card? this would be the cheapest *enhancement* :) .. But this might influence the running applications under test in a bad manner? If not, I would do this. – hek2mgl Jul 03 '14 at 10:58
  • 1
    Oh, I forgot, it's Linux! Of course you can just slow down your network card instead of purchasing a slower one: http://www.cyberciti.biz/faq/linux-change-the-speed-and-duplex-settings-of-an-ethernet-card/ – hek2mgl Jul 03 '14 at 11:00
  • @hek2mgl Which speed should I set for my 52 000 msg/sec * 500 bytes = 25 MB/sec = 200 MBit/sec ? – Neighbour Jul 07 '14 at 07:20
  • 1
    I would start with trying 100BaseT. If the kernel still drops packages, then you can use 10BaseT which is even slower. – hek2mgl Jul 07 '14 at 07:34
  • Or you could just increase the buffer size:http://unix.stackexchange.com/questions/18311/buffer-size-for-capturing-packets-in-kernel-space – Peter Brittain Mar 19 '16 at 00:40

0 Answers0