I have two virtualization servers based on libvirt and KVM. And sometimes I see packages get lost on one particular virtual machine. After restarting the virtualization host this problem is solved, but it helps for a while.
I have made filters in iptraph-ng and I see that packets are lost between bridge interfaces.
You can see it here::
Fri Mar 3 00:21:43 2023; ICMP; ens3f0; 84 bytes; from 10.10.10.68 to 10.10.11.23; echo req
Fri Mar 3 00:21:43 2023; ICMP; bond1; 84 bytes; from 10.10.10.68 to 10.10.11.23; echo req
Fri Mar 3 00:21:43 2023; ICMP; bond1.121; 84 bytes; from 10.10.10.68 to 10.10.11.23; echo req
Fri Mar 3 00:21:43 2023; ICMP; ens3f0; 84 bytes; from 10.10.10.68 to 10.10.11.23; echo req
Fri Mar 3 00:21:43 2023; ICMP; bond1; 84 bytes; from 10.10.10.68 to 10.10.11.23; echo req
Fri Mar 3 00:21:43 2023; ICMP; bond1.121; 84 bytes; from 10.10.10.68 to 10.10.11.23; echo req
When there is no packet loss:
Fri Mar 3 00:21:43 2023; ICMP; ens3f0; 84 bytes; from 10.10.10.68 to 10.10.11.23; echo req
Fri Mar 3 00:21:43 2023; ICMP; bond1; 84 bytes; from 10.10.10.68 to 10.10.11.23; echo req
Fri Mar 3 00:21:43 2023; ICMP; bond1.121; 84 bytes; from 10.10.10.68 to 10.10.11.23; echo req
Fri Mar 3 00:21:43 2023; ICMP; vnet18; 84 bytes; from 10.10.10.68 to 10.10.11.23; echo req
Fri Mar 3 00:21:43 2023; ICMP; vnet18; 84 bytes; from 10.10.11.23 to 10.10.10.68; echo rply
Fri Mar 3 00:21:43 2023; ICMP; bond1.121; 84 bytes; from 10.10.11.23 to 10.10.10.68; echo rply
Fri Mar 3 00:21:43 2023; ICMP; bond1; 84 bytes; from 10.10.11.23 to 10.10.10.68; echo rply
Fri Mar 3 00:21:43 2023; ICMP; ens3f0; 84 bytes; from 10.10.11.23 to 10.10.10.68; echo rply
Settings of virtual interface:
virsh dumpxml vm-rep |xmllint -xpath ///interface -
<interface type="bridge">
<mac address="52:54:00:2b:35:b4"/>
<source bridge="br121"/>
<target dev="vnet18"/>
<model type="virtio"/>
<alias name="net0"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
When packets are lost, I don't see any RX/TX errors or drops on the bridge interface, bonded vlan interface or tap interface (ifconfig br121 and etc.). I don't see any increase in counters with virsh domifstat vm-rep vnet18. I switched from linuxbridge to openvSwitch, but that didn't help, and if I run ovs-ofctl dump-ports ovsbr121, I don't see any errors or drops. I enabled the debug level in openvSwitch and I also don't see any errors or drops in the bridge interfaces.
This virtual machine is a repository on nginx and has a block device rbd on Serh and no memory/processor/block device load. The virtualization host is also not loaded, in top I do not see high values in wa, hi, si, st or in the load average value. Idle is almost always 96. There are only 17 virtual machines on the host.
In the logs on the virtualization hosts and VM, I saw no reason for this behavior. I don't understand how I can figure out the cause of packet loss. It looks like a short-term VM freeze.
Environment:
- Rocky Linux 8.5 on virtualization host and VM.
- libvirtd (libvirt) 6.0.0
- QEMU emulator version 4.2.0 (qemu-kvm-4.2.0-60.module+el8.5.0+772+743bb337.2)
- kernel 4.18.0-348.23.1.el8_5.x86_64
- virt-install 2.2.1
The VM was created with virt-install:
virt-install \
--machine q35 \
--name vm-rep \
--memory=16384 \
--vcpus=8 \
--os-variant rocky8.5 \
--disk size=20,bus=virtio \
--network bridge=br121,model=virtio \
--graphics=vnc \
--boot hd,cdrom \
--noautoconsole \
--autostart \
--channel unix,mode=bind,path=/var/lib/libvirt/qemu/vm-rep.agent,target_type=virtio,name=org.qemu.guest_agent.0 \
--location /var/lib/libvirt/isos/Rocky-8.5-x86_64-minimal.iso \
--initrd-inject=ks.cfg \
--extra-args "inst.ks=file:/ks.cfg console=ttyS0,115200n8"
Sometimes such packet losses occur to other VMs, but this one most often. Rebooting the VM does not help, only rebooting the virtualization host.
Is there any way to trace/dump or at least somehow see what happens to lost packets?
