I have a small Ubuntu x64 server, which provides dhcp, naming, file and such services to my home network. It's simply connected to unmanaged SOHO switch laying next to it and just works. But, it has unused 4xGbE i350 NIC card in PCIe slot and I had this idea of using this NIC instead of the current switch. It probably won't have any advantages if it will be also plain unmanaged switch like the current dedicated device but I wanted to learn something (and I could maybe turn it into managed switch later, maybe implement some IPS/IDS and further stuff). How I was surprised to find out I have no idea how. I did my share of searching of course, but it left me confused even more. First of all, why all howtos and articles configure bridges to achieve switching? With unmanaged switch all ports are equal, has no IPs and one of them leads to gateway. Is that even possible to create that on Linux server (ignoring it may not be really useful considering low price of a small switch)?
-
2All of the howtos and documents configure a bridge because that is exactly how you do it. – Michael Hampton Mar 12 '21 at 02:07
-
Clearly. But that was my question, is it the only possible way? Bridged iface has an IP address and requires ip forwarding. While it mostly provides the comparable service, it is not the same thing. – Ren Mar 12 '21 at 08:12
-
1What do you mean by that? A bridge does not require either an IP address or IP forwarding. Though virtually everyone actually _does_ use IP on their networks. I would be very surprised if you were an exception, since all of the services you mention use IP. – Michael Hampton Mar 12 '21 at 08:35
-
There's clearly something about the concept I do not understand. Small switch uses no IP addresses, it's completely transparent and does all the work on L2, working simply with MAC addresses. Now, when I go and configure Linux bridge across multiple ports, I get a a new, virtual interface. It has an IP address and I believe it needs routing/kernel IP forwarding. If that's true, they are basically very different devices (though providing comparable service). As I said, it is not about making something work, it's about understanding how things work inside so that difference is important. – Ren Mar 12 '21 at 16:58
-
Your Linux bridge only has an IP address because you chose to give it one. It is still not required. – Michael Hampton Mar 12 '21 at 19:47
-
I see, it makes sense. I'm gonna move mandatory services to another box and will do a few experiments with various settings and tcpdump. Thanks for helping me find a direction. – Ren Mar 14 '21 at 16:44
1 Answers
Use the bridge driver.
The bridge is a software implementation of a Layer 2 switch, where each interface added into the bridge is a switchport.
Let's say you have your 4 i350 ports in bridge br0 with two other PCs plugged in:
[ Remote PC A - MAC aa:aa:aa:aa:aa ] ---- [ i350 #1 ] ---- .------------.
[ i350 #2 ] ---- | Linux PC |
[ i350 #3 ] ---- | bridge br0 |
[ Remote PC B - MAC bb:bb:bb:bb:bb ] ---- [ i350 #4 ] ---- '------------'
Remote PC A tries to reach Remote PC B.
PC A will ARP out to the broadcast destination MAC address (all ff) with its own source MAC (all aa).
The bridge will receive that ARP Request and learn that MAC aa is down bridgeport i350 #1.
The bridge then forwards that broadcast out all the other i350 ports, because that is the correct behaviour when a switch recieves a broadcast packet.
Remote PC B receives the ARP, and generates a reply to PC A with destination MAC of PC A (all aa) and its own source MAC (all bb).
The bridge will receive that ARP Reply and learn that MAC bb is down bridgeport i350 #4.
The bridge sees the destination MAC is aa, and the bridge knows that MAC aa is available down port i350 #1 and so sends the frame out that NIC.
You can add an IP address to a bridge and use it like a regular network interface, route with it, NAT with it, etc but you don't have to.
The above happens even if kernel tunable net.ipv4.ip_forward=0 is set, because the bridge driver is not performing IP forwarding or IP routing or IP NAT, it is performing Layer 2 packet switching like any other unmanaged Layer 2 switch.
You can also run Spanning Tree on the bridge if you are connecting it to a larger network.
You said you're using Ubuntu. The following should work for Ubuntu 20.04 using NetworkManager:
/etc/NetworkManager/system-connections/br0.nmconnection
[connection]
id=br0
type=bridge
interface-name=br0
permissions=
[bridge]
stp=false
[ipv4]
dns-search=
ignore-auto-dns=true
ignore-auto-routes=true
method=disabled
never-default=true
[ipv6]
addr-gen-mode=stable-privacy
dns-search=
ignore-auto-dns=true
ignore-auto-routes=true
method=ignore
never-default=true
[proxy]
Repeat this file for each i350 named netX, using each NIC's correct MAC address:
/etc/NetworkManager/system-connections/netX.nmconnection
[connection]
id=netX
type=ethernet
interface-name=netX
master=br0
metered=2
permissions=
slave-type=bridge
[ethernet]
mac-address=xx:xx:xx:xx:xx:xx
mac-address-blacklist=
[bridge-port]
- 3,536
- 21
- 29
-
This is encyclopedic quality answer, thanks for that. I managed to get it working since my original post but it still waits for some tests, I'm really curious about latencies, throughput of multiple traffic etc. in comparison with a simple cheap switch (I expect the switch to have a slight edge over i350/bridge based switching but we'll see). – Ren Mar 20 '21 at 20:28
-
There's an old research paper from 2004 which found a PC running the bridge was comparable in throughput and latency to a Cisco 2950, so I expect a PC these days would be able to outpace a typical $5 unmanaged switch. I use bridges a lot at work for testing between VMs, and I'm easily able to do RAM bandwidth for a single TCP stream like iperf. The performance of the bridge is a lot better than the performance of IP routing. – suprjami Mar 21 '21 at 02:18
-
Now that would be awesome.:-) I'm superbusy these days but I'll try to share my findings here as soon as I get some. – Ren Mar 21 '21 at 10:44