I am working on an application that uses DPDK to write packet payloads to file and am investigating whether or not PcapPlusPlus could be used for this purpose. My setup is as follows:
I am using a Mellanox ConnectX-5 NIC, Ubuntu 22.04.01, and the latest DPDK and PcapPlusPlus. Kernel: 5.15.0-53-generic and 10 1GB hugepages.
Mellanox Connectx-5 NIC:
Device #1:
----------
Device Type: ConnectX5
Part Number: MCX516A-CCA_Ax
Description: ConnectX-5 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
PSID: MT_0000000012
PCI Device Name: /dev/mst/mt4119_pciconf0
Base GUID: b8cef60300fb1330
Base MAC: b8cef6fb1330
Versions: Current Available
FW 16.35.1012 N/A
PXE 3.6.0804 N/A
UEFI 14.28.0015 N/A
Status: No matching image found
Running ethtool on the card:
Settings for ens7f0np0:
Supported ports: [ Backplane ]
Supported link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
50000baseCR2/Full
50000baseKR2/Full
100000baseKR4/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None RS BASER
Advertised link modes: 100000baseKR4/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: RS
Speed: 100000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
netlink error: Operation not permitted
Current message level: 0x00000004 (4)
link
Link detected: yes
Here is my issue. In the diagram, each link consists of two sets of packets. One set from port 0 and one set from port 1 etc. If using tcpdump I can record the packets from a single port without issue to a RAID0 array consisting of 4 NVME drives.
sudo tcpdump -i ens7f0np0 udp -nn -# -N -B 1048576 -t -q -Q in -p -w /mnt/md0/test/test.pcap dst port 50340
I get a .pcap file that is 2GB in size for a 1 second test as expected. Two ports is too much for the kernel so If I try to record everything I get dropped packets. This is where DPDK comes in. I wanted to use the DPDK filter example to do the same test with DPDK.
sudo ./DpdkTrafficFilter -d 2 -f /mnt/md0/ -o 'UDP' -c 7
I run the example and here is the output:
EAL: Detected CPU lcores: 56
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available 2048 kB hugepages reported
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:17:00.0 (socket 0)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:17:00.1 (socket 0)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:31:00.0 (socket 0)
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:31:00.1 (socket 0)
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:b1:00.0 (socket 1)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:b1:00.1 (socket 1)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:ca:00.0 (socket 1)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:ca:00.1 (socket 1)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:e3:00.0 (socket 1)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:e3:00.1 (socket 1)
mlx5_net: No available register for sampler.
TELEMETRY: No legacy callbacks, legacy socket not created
Opened device #2 with 1 RX queues and 1 TX queues. RSS hash functions:
RSS_IPV4
RSS_IPV6
Using core 1
Core configuration:
DPDK device#2: RX-Queue#0;
Using core 2
Core configuration:
None
^C
Application stopped
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Core ID | Packet Cnt | Eth Cnt | ARP Cnt | IPv4 Cnt | IPv6 Cnt | TCP Cnt | UDP Cnt | HTTP Cnt | Matched TCP Flows | Matched UDP Flows | Matched Packets |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| 1 | 514051 | 514051 | 0 | 514051 | 0 | 0 | 514051 | 0 | 0 | 1 | 514050 |
| 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Total | 514051 | 514051 | 0 | 514051 | 0 | 0 | 514051 | 0 | 0 | 1 | 514050 |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
However, the pcap file is only 725MB for a 1 second test! Any idea what I am doing wrong? Are packets being dropped? If so why is the performance worse than tcpdump?
Also, not sure if it matters, but the traffic I am receiving is MTU 9000. I did change the DpdkTrafficFilter code to set the mtu to 9000 but got the same results.
