1

I am trying to use xdp to redirect network traffic between VM taps. I notice that when I load the xdp program, the network is much slower (130 Mbits/sec) from when I am loading the same code on the vmbr (4.5 Gbits/sec). I cannot understand what cause this speed performance to go down. This is how network architecture looks like: enter image description here Note: the redirect itself works, just can't understand why the speed is much slower.

TylerH
  • 20,799
  • 66
  • 75
  • 101
Lidorelias3
  • 27
  • 1
  • 10
  • Did you try to retrieve perf samples and try to compare the functions that contribute to the higher/lower CPU? – pchaigno Apr 05 '21 at 13:23
  • What do you mean by comparing the functions? I didn’t try that – Lidorelias3 Apr 06 '21 at 14:04
  • See http://www.brendangregg.com/perf.html. – pchaigno Apr 06 '21 at 14:26
  • @Lidorelias3 can you please confirm you are loading XDP in `br0` and getting 4.5Gbps and when you load XDP on `bond0` you are getting 130Mbps? If this is right understand please update. Second point, XDP works only in RX mode only and no TX. Hence at `br0` RX traffic includes VM traffic and bon0 traffic. But at `bond0` RX include `eht0, eth1` and `br0` which needs to comes from kernel stack (bond0) . – Vipin Varghese Apr 21 '21 at 03:33
  • @VipinVarghese Yes, this is right understood. I find out that it all connects to hardware offloads. The net stack wants to offload the packet to the vmbr(br0) and it cant reach there because I redirecting it to the other tap. so things like checksum offload, TCP Segmentation Offload, and Large Recv Offload are not actually made those offload. For the checksum part, I managed to solve it. but for the others, I am starting to think that impossible in XDP – Lidorelias3 Apr 22 '21 at 15:13
  • @Lidorelias3 thanks for the update, XDP is supported in RX only there is no TX XDP packet intercept. One can redirect the packet but will need to perform checksum LSO and filtering if it is not coming from NIC which does the same inherently. – Vipin Varghese Apr 23 '21 at 02:42
  • @VipinVarghese Thanks for your answer. I know that XDP is supported only on the RX side. about the checksum LSO, how can I perform it on XDP? – Lidorelias3 Apr 24 '21 at 10:47
  • @Lidorelias3 through software only, as XDP sits before the actual Network layer. – Vipin Varghese Apr 25 '21 at 07:11
  • @VipinVarghese but LSO needs to wait for data before it adding it together and send to the other side. In XDP is it possible to wait and not doing anything until we want to send the large data? – Lidorelias3 Apr 25 '21 at 11:20
  • @Lidorelias3 can you please share the XDP code you are binding with br0 and steps to reproduce the performance issue? – Vipin Varghese Apr 27 '21 at 02:40
  • @VipinVarghese Unfortunately not. But my code just takes the packet, calculates the checksum because of checksum offload, and redirects to the other tap. The performance issue caused because the LSO and TSO that there are on VirtIO. I also notice that with the XDP program that locates on the tap who just prints the packet size and passes it I got a packet with 65500 bytes on size (I think this is possible only because LSO). with XDP redirect I only get packets at size 1500. – Lidorelias3 Apr 27 '21 at 13:29
  • @Lidorelias3 may i request a live debug, as I believe I might be missing some understanding from the current discussion. Shall we sync up via skype, hangout? – Vipin Varghese May 10 '21 at 15:53
  • @VipinVarghese can I send you a private message on telegram or some messaging app? – Lidorelias3 May 15 '21 at 15:52
  • i do not use telegram, you can send your message on skype, hangout or email. – Vipin Varghese May 16 '21 at 06:08
  • Okay, let's try hangout. Can you send me your email? – Lidorelias3 May 16 '21 at 09:12
  • you can reach me with id vipinpv85 – Vipin Varghese May 18 '21 at 14:44
  • @Lidorelias3 I ahve not recieved any invite or share in hangout or skype – Vipin Varghese May 20 '21 at 15:16
  • @VipinVarghese I send you a message to hangouts. I used the email vipinpv85@gmail.com. Is that correct? – Lidorelias3 May 21 '21 at 23:02
  • I have not received email yet. What is email id you have send from? – Vipin Varghese May 22 '21 at 06:07
  • My email is lidorelias3@gmail.com – Lidorelias3 May 22 '21 at 12:04
  • I have not receieved any email, but will add you to the list – Vipin Varghese May 24 '21 at 03:20
  • did a live chat (since code can not be exposed) with @Lidorelias3. Mentioned possible reason for performance drop is related to 1) bridge kernel thread, 2) vcpu pining to host, 3) memory pinning and 4) virtio-net incorrect offload. I have tried to communicate if VIRTIO-NET to test with disabling `ethtool -K eth0 gro off gso off`. But as per Lideorelias3 observation without XDP virtio does LRO and GSO offload, with XDP LRO and GSO offload, is not happening because packets are intercepted by TAP XDP. Hence further debugging was not possible. – Vipin Varghese May 27 '21 at 08:19
  • @VipinVarghese Thanks for your time! Also mentioning that when disabling gro gso also tso etc... via ethtool my performance with XDP is good in relation to without XDP, but still slower then when all the offloading are enabled – Lidorelias3 May 27 '21 at 23:13
  • 1
    @Lidorelias3 thanks for the update that is good news for me as it matches my observation in my setup. With respect to XDP performance as I have suggested, it depends upon the code logic (other than redirect), whether redirect or redirect_map is used, separate host CPU cores pinned for Tap1 and Tap2 for performance. But since it closed source code and debug access to the environment is not available, it is not easy for me to debug. – Vipin Varghese May 28 '21 at 01:13

0 Answers0