The overlay packets have to traverse a complex path in the kernel (involving multiple network stacks) before going out to the wire. When I trace the main kernel functions involved (using eBPF), the path of a packet looks something like this:
function device
======== ======
sendto veth0
sock_sendmsg veth0
udp_sendmsg veth0
udp_send_skb veth0
ip_send_skb veth0
ip_local_out veth0
veth_xmit veth0
do_softirq (softirq handling)
net_rx_action (softirq handling)
process_backlog (softirq handling)
netif_receive_skb vxlan0
br_forward vxlan0
vxlan_xmit vxlan0
udp_tunnel_xmit_one eno1
ip_tunnel_xmit eno1
ip_local_out eno1
ixgbe_xmit_frame eno1
Here, veth0
is the virtual interface in the overlay network, vxlan0
is the VxLAN overlay bridge, and eno1
is the physical interface.
So my question is that... since the sender doesn't know anything about the overlay network stuff (encapsulation, virtual network stacks, etc), does the sender's (running in the overlay network) sendto()
return after sending the packet out of veth0
? Or does it only return after the packet has left eno1
?
I'm confused because it involves raising a softirq in the middle and softirqs are processed in a separate context/thread (but on the same core by default). I'm interested in this because I'm researching on improving the performance of overlay networks and I want to see if moving the softirq handling parts to a different cpu core can help.