2

I've setup a pfsense guest on KVM and and trying to pass tagged and untagged traffics to the VM for obvious reasons. My setup looks like this -->

The KVM has 2 NICs(both virtio type):

  • 1st(for untagged and tagged traffics): macvtap(vepa) linked to enp1s0f0.
  • 2nd(for untagged data only): macvtap(passthrough) linked to enp1s0f3.
  • Wire1(untagged 10 + tagged vlan 28) ==> enp1s0f0 of the host.
  • Wire2(untagged) ==> enp1s0f3 of the host.

VM has 2 static IP configured 10.10.10.1 and 10.10.28.1 and DHCP on 2nd NIC.

Now, the problem is : all the traffic over untagged vlan is going through in and out without problem, but tagged traffic is completely lost somewhere. i.e. ping between 10.10.10.1 and other host on same vlan works easy but 10.10.28.1 is not reachable at all.

Now I tried Wireshark promiscuous mode on enp1s0f0 and when Wireshark is running, tagged data passing through. Finally found, promiscuous mode makes the difference to pass tagged data, but that cant be a permanent solution.

I read lots of wikis and blogs, none has clear setup config for similar requirements. I'm not sure where I'm going wrong with the setup?

Andrew Schulman
  • 8,811
  • 21
  • 32
  • 47
amlanhldr
  • 31
  • 4
  • It looks similar to: https://serverfault.com/questions/1026876/trunking-802-1q-inside-kvm-via-macvtap (which has no answer for now). – A.B Oct 30 '20 at 20:58
  • I would use a normal bridge in your situations, tagged frames work fine over that. – Tero Kilkanen Oct 30 '20 at 22:57
  • Thanks A.B . Thanks @TeroKilkanen for the pointer. I found couple of config using bridge, but they suggest individual VLAN to individual VM NIC. I need all the tagged + untagged frames on one NIC. Sorry, I'm pretty novice on KVM networking. – amlanhldr Oct 31 '20 at 10:24
  • You can add a whole Ethernet interface to the bridge, and all untagged and tagged frames will reach the VM. – Tero Kilkanen Oct 31 '20 at 10:37
  • Asking the same question I asked in the other Q/A: Did you change the MAC address of the interface (enp1s0f0 if I followed correctly) from the one which was inherited? If you don't change it, does the behaviour change? Of course using a bridge would work, but I'm curious about this issue. – A.B Oct 31 '20 at 15:39
  • Hi A.B, I didn't change any MAC. BTW, I'm trying the bridge, and I see both tagged and untagged traffic on the bridge interface without promiscuous mode, but for some reason no traffic is not passing through to the VM. trying the figure that out.... – amlanhldr Oct 31 '20 at 16:35
  • while both tagged and untagged frames went through while promiscuous mode is on with *macvtap* driver.. – amlanhldr Oct 31 '20 at 16:37
  • again about the MAC: you have to verify that the MAC address on enp1s0f0 in the VM is the same as the MAC address on the macvtap device on the host. Even if *you* didn't change the MAC doesn't mean the configuration didn't change it. – A.B Oct 31 '20 at 18:08
  • Found these from the host:: enp1s0f0: link/ether 2c:53:4a:09:8f:86 brd ff:ff:ff:ff:ff:ff/ enp1s0f3: link/ether 52:54:00:10:c0:af brd ff:ff:ff:ff:ff:ff/ macvtap0@enp1s0f3: link/ether 52:54:00:10:c0:af brd ff:ff:ff:ff:ff:ff/ macvtap1@enp1s0f0: link/ether 52:54:00:e0:8f:cd brd ff:ff:ff:ff:ff:ff. I see, MAC of enp1s0f3 has been changed because **macvtap0:passthrough** and **macvtap1:bridge** ?? The MAC address inside the VM are same as of the macvtap interfaces on host. – amlanhldr Oct 31 '20 at 19:28
  • Never mind your settings are fine. I was reading a bug but I realize that wasn't the same description anyway. So that wasn't your case in that bug ( https://bugzilla.redhat.com/show_bug.cgi?id=1531459 ) – A.B Oct 31 '20 at 19:48
  • Thanks @A.B for the lead, I'll check that. – amlanhldr Oct 31 '20 at 19:48

2 Answers2

0

Well, after couple of tries, I'm able to figure out an workaround. I followed the bridge way and that finally worked. I'm in a kind of impression, this is way better that leaving a port promiscuous, since that'll increase CPU overhead.

In case someone followed through here, leaving the quick guide to setup the bridge: Create a bridge --> add the eth port to the bridge --> add Manual IP to the bridge --> set the bridge interface as KVM nic.

I'll still look for macvtap way. If anyone have any insight, thank you for sharing..

amlanhldr
  • 31
  • 4
0

After doing quite some testing I found the following working options for tagged traffic in a KVM guest:

bridge

Well known and documented approach. Creates more load compared to the other alternatives.

SR-IOV

Quite some steps required to set this up. Here a slice of the real NIC hardware is passed as virtual function to the guest. The guest uses a native driver, and has full functionality. Including tagging/untagging using the OS native functions.

CPU, board/bios and the NIC hardware has to support this. Got this actually working on a hp z420 with intel i350-t4.

Further reading: https://software.intel.com/content/www/us/en/develop/articles/configure-sr-iov-network-virtual-functions-in-linux-kvm.html

macvtap

Tagging inside the guest did also not worked out for me. Creating the tagged links on the host (e.g. eth0.100), and passing these like an untagged link via macvtap into the guest makes them show up there as an independent link.

These links behave then as if they where connected to an access port on a VLAN capable switch. I can use them untagged in the guest, the VM host performs tagging/untagging, and the network sees tagged traffic.

conclusion

All three alternatives give near-native 1 GBe performance. Bridge with more CPU overhead on my system (1.3 vs 0.8 cores), macvtap and sr-iov performed quite the same.

Tagging on the host and passing via macvtap disables the guest from freely choosing the VLANs, so tagging and physical port routing is kept on the host. That's the way I'll connect my guests for now.

For 10 GBe + IMHO sr-iov is the only feasible approach, however.

FreeJack
  • 121
  • 3
  • Unfortunately, this macvtap option doesn't work for me, cause my guest is a pfsense router, I have to have vlans routing inside guest. But thanks for the consize and useful info. :) – amlanhldr Mar 30 '21 at 02:22
  • For me it works like a charm virtualizing pfsense to route between vlans -- the pfsense use case was the main reason for me doing the research how to get efficient NIC virtualization. The tagged devices appear as interfaces in pfsense, and are used there without further tagging. – FreeJack Mar 31 '21 at 03:28
  • I think I'm getting you idea.. amazing one. will definitely give a try when I'm home... – amlanhldr Apr 01 '21 at 04:26
  • Have also a look at the trustGuestRxFilters="yes" setting -- the default "no" causes very weird errors, cause multicast is not working. Took me another hour debugging... https://superuser.com/questions/944678/how-to-configure-macvtap-to-let-it-pass-multicast-packet-correctly – FreeJack Apr 01 '21 at 16:14