1

note: I still have been unable to get this config to work, but will post my solution as soon as I figure it out.

I am setting an Ubuntu host server (16.04) with a number of KVMs running on it. The host server has 4 physical NICs which are bonded in pairs to access a public facing network, and a private network. The host has a primary public subnet assigned to it (with only one usable external IP) and a primary private subnet (with one usable internal IP). Additionally I have acquired two additional 8 IP subnets, one public and one private for the virtual machines on this host. For example here is my IP layout:

FOR HOST:

  • Public Primary Subnet: 200.0.0.72/29 - Usable Public IP: 200.0.0.75
  • Private Primary Subnet: 10.0.0.128/26 - Usable Private IP: 10.0.0.138

FOR VMS

  • Public Portable Subnet: 200.0.1.240/29, 5 Usable Public IPs
  • Private Portable Subnet: 10.0.1.216/29, 5 Usable Private IPs

I am having a lot of trouble getting the host to have a separate subnet from my bridge interfaces for the VMs. It seems I can't get the host to network correctly when I have bridges specified, and bridges only work, when I do not specify a network in the host area on the bonded interface. My interfaces file is below. What you see here is my "desired" outcome, and doesn't actually work. I've been able to get pieces to work by commenting different portions and restarting networking. I also added routing commands (route add) per instructions from my ISP, but they have not been able to help with the bridging aspect and the VMs. 

############################################
#For more information, see interfaces(5).
############################################
##Loopback
auto lo
iface lo inet loopback

## Setup bond0 interface - INTERNAL CONNECTIONS
auto bond0
iface bond0 inet static
bond-lacp-rate 1
#pre-up /sbin/ethtool -s bond0 speed 100 duplex full autoneg off
post-up ifenslave bond0 eth0 eth2
pre-down ifenslave -d bond0 eth0 eth2
address 10.0.0.138
netmask 255.255.255.192
gateway 10.0.0.129
bond-slaves none
bond-mode 4
bond-lacp-rate fast
bond-miimon 100
bond-downdelay 0
bond-updelay 0
bond-xmit_hash_policy 1
up route add -net 10.0.0.0 netmask 255.0.0.0 gw 10.0.0.129

auto eth0
iface eth0 inet manual
bond-master bond0
#pre-up /sbin/ethtool -s eth0 speed 100 duplex full autoneg off

auto eth2
iface eth2 inet manual
bond-master bond0
#pre-up /sbin/ethtool -s eth2 speed 100 duplex full autoneg off

auto br0
iface br0 inet static
address 10.0.1.218
netmask 255.255.255.192
gateway 10.0.1.217
bridge_ports bond0
bridge_stp off
bridge_fd 9
bridge_hello 2
bridge_maxage 12

## Setup bond1 interface - EXTERNAL CONNECTIONS
auto bond1
iface bond1 inet static
bond-lacp-rate 1
post-up ifenslave bond1 eth1 eth3
pre-down ifenslave -d bond1 eth1 eth3
bond-slaves none
bond-mode 4
bond-lacp-rate fast
bond-miimon 100
bond-downdelay 0
bond-updelay 0
bond-xmit_hash_policy 1
address 200.0.0.75
netmask 255.255.255.248
gateway 200.0.0.73
up route add -net 0.0.0.0 netmask 0.0.0.0 gw 200.0.0.73

auto eth1
iface eth1 inet manual
bond-master bond1

auto eth3
iface eth3 inet manual
bond-master bond1

auto br1
iface br1 inet static
address 200.0.1.242
netmask 255.255.255.248
gateway 200.0.1.241
#up iptables-restore < /etc/iptables.rules
post-up echo 0 > /sys/devices/virtual/net/br1/bridge/multicast_snooping
bridge_ports bond1
bridge_stp off
bridge_fd 9
bridge_hello 2
bridge_maxage 12

If I already have a subnet on my host (on the bond interface), I may be wrong in specifying a specific address on the bridged interface, but then Im not sure how to specify a separate subnet for the VMS (on bridge) without doing that. Thanks in advance for any comments/suggestions as to what Im doing wrong here.

#Addition 1: VLANS

Below are the subnets for the VLANs given to me by my ISP. The "Primary IP for future server only" is for a primary IP given to another customer/host, and not accessible by me. Additionally there are two primary private IPs, one is accessible to my host OS and the other is a management interface for console access (via the bios it looks like).

There are two VLANs, one public and one private, and they each are numbered and have a router associated with them. I.e VLAN number 1200 router zzz02a.bbb03 for public network, and a similar one for private.

Public Primary Subnet

200.0.0.72  Reserved    Network
200.0.0.73  Reserved    Gateway
200.0.0.74  Reserved    Primary IP for future server only
200.0.0.75  In Use  host.mydomain.com
200.0.0.76  Reserved    Primary IP for future server only
200.0.0.77  Reserved    Primary IP for future server only
200.0.0.78  Reserved    Primary IP for future server only
200.0.0.79  Reserved    Broadcast

Public Portable Subnet

200.0.1.240 Reserved    Network
200.0.1.241 Reserved    Gateway
200.0.1.242 Available   vm1.mydomain.com
200.0.1.243 Available   <<available for vm>>
200.0.1.244 Available   <<available for vm>>
200.0.1.245 Available   <<available for vm>>
200.0.1.246 Available   <<available for vm>>
200.0.1.247 Reserved    Broadcast

Private Primary Subnet 

10.0.0.128  Reserved    Network
10.0.0.129  Reserved    Gateway
10.0.0.130  Reserved    Primary IP for future server only
10.0.0.131  Reserved    Primary IP for future server only
10.0.0.132  Reserved    Primary IP for future server only
10.0.0.133  Reserved    Primary IP for future server only
10.0.0.134  Reserved    Primary IP for future server only
10.0.0.135  Reserved    Primary IP for future server only
10.0.0.136  Reserved    Primary IP for future server only
10.0.0.137  Reserved    Primary IP for future server only
10.0.0.138  In Use  host.mydomain.com
10.0.0.139  Reserved    Primary IP for future server only
10.0.0.140  Reserved    Primary IP for future server only
10.0.0.141  Reserved    Primary IP for future server only
10.0.0.142  Reserved    Primary IP for future server only
10.0.0.143  Reserved    Primary IP for future server only
10.0.0.144  Reserved    Primary IP for future server only
10.0.0.145  Reserved    Primary IP for future server only
10.0.0.146  Reserved    Primary IP for future server only
10.0.0.147  Reserved    Primary IP for future server only
10.0.0.148  Reserved    Primary IP for future server only
10.0.0.149  In Use  host.mydomain.com
10.0.0.150  Reserved    Primary IP for future server only
10.0.0.151  Reserved    Primary IP for future server only
10.0.0.152  Reserved    Primary IP for future server only

Private Portable Subnet 

10.0.1.216  Reserved    Network
10.0.1.217  Reserved    Gateway
10.0.1.218  Available   vm1.mydomain.com
10.0.1.219  Available   <<available for vm>>
10.0.1.220  Available   <<available for vm>>
10.0.1.221  Available   <<available for vm>>
10.0.1.222  Available   <<available for vm>>
10.0.1.223  Reserved    Broadcast
hmorton
  • 13
  • 4
  • How exactly are these single IP addresses and /29 subnets delivered to you by your ISP? Are they delivered in separate VLANs? – Tero Kilkanen Mar 25 '17 at 02:43
  • Yes, VLANs, primary VLANs are shared with other servers/customers, and I have one of the addresses, and the portable /29 VLANs are all mine. – hmorton Mar 25 '17 at 07:03
  • Are all the VLANs tagged or is the primary VLAN default/untagged VLAN? – Tero Kilkanen Mar 25 '17 at 12:38
  • Can you define tagged vs untagged VLANs? Im not sure what those are. – hmorton Mar 25 '17 at 21:18
  • A tagged VLAN is a VLAN with a VLAN header, which contains the VLAN ID, which is between 1 and 4095, while untagged VLAN has no VLAN header. – Tero Kilkanen Mar 26 '17 at 01:06
  • Actually now that I read the question closer, I realized these networks are delivered via separate ethernet bonds. – Tero Kilkanen Mar 26 '17 at 01:17
  • Can you clarify, if those private subnetworks are supposed to traverse over the bonds to the switch? If that is the case, you need to have separate subnets set up for them, using VLANs. – Tero Kilkanen Mar 26 '17 at 01:22
  • Yes - I think the private and public networks are on separate switches, because I bond the two hardware interfaces together (eth0 and eth2 ) for the private subnets and the other two hardware interfaces (eth1 and eth3) for the public subnets. They are called VLANs in the administrator portal of the ISP. – hmorton Mar 26 '17 at 17:11
  • There needs to be total four separate broadcast domains in order to get those four separate IP subnets from the switch to the server. Broadcast domains can be made with physical connections or VLANs. So, you need to find out the exact VLAN configuration on the switches your servers are connected to in order for us to provide a working configuration. – Tero Kilkanen Mar 26 '17 at 22:21
  • Hi Tero, I just posted a more complete description of the VLANs and their IPs. Let me know if this is what you were looking for? It appears that I have two VLANs association with my service, one for the public network (with the two subnets), and the other for the private network (with its two subnets). Thanks again for your help. – hmorton Mar 27 '17 at 07:27
  • It is still not clear if they are actually Ethernet level VLANs or what. That is the crucial information needed here. That is, are the portable and primary subnets separate networks. – Tero Kilkanen Mar 27 '17 at 09:40
  • Tero, I think they are but I can check with my ISP. I have obfuscated the actual addresses a little bit, in reality the portable and primary subnets are very different from each other i.e they appear to be on completely different class B ranges – hmorton Mar 27 '17 at 10:57
  • VLANs are trunk mode if that helps. – hmorton Mar 28 '17 at 12:43
  • You still need the VLAN IDs assigned to each subnet, without that you cannot configure the network interfaces correctly. – Tero Kilkanen Mar 28 '17 at 12:45

1 Answers1

0

I think there are no vlans. If you want to share all 4 subnets​ to your virtual machines you should assign them on the bridges. The interfaces file should be:

############################################
#For more information, see interfaces(5).
############################################
##Loopback
auto lo
iface lo inet loopback

## Setup bond0 interface - INTERNAL CONNECTIONS
auto bond0
iface bond0 inet manual
bond-slaves eth0 eth2
bond-mode 4
bond-lacp-rate fast
bond-miimon 100
bond-downdelay 0
bond-updelay 0
bond-xmit_hash_policy 1

auto eth0
iface eth0 inet manual
bond-master bond0

auto eth2
iface eth2 inet manual
bond-master bond0

## Setup VLAN ID 1201 on bond0 interface
auto bond0.1201
iface bond0.1201 inet manual
vlan_raw_device bond0

auto br0
iface br0 inet static
address 10.0.0.138
netmask 255.255.255.192
bridge_ports bond0.1201
bridge_stp off
up route add -net 10.0.0.0 netmask 255.0.0.0 gw 10.0.0.129

auto br0:1
iface br0:1 inet static
address 10.0.1.218
netmask 255.255.255.192

## Setup bond1 interface - EXTERNAL CONNECTIONS
auto bond1
iface bond1 inet manual
bond-slaves eth1 eth3
bond-mode 4
bond-lacp-rate fast
bond-miimon 100
bond-downdelay 0
bond-updelay 0
bond-xmit_hash_policy 1

auto eth1
iface eth1 inet manual
bond-master bond1

auto eth3
iface eth3 inet manual
bond-master bond1

## Setup VLAN ID 1200 on bond1 interface
auto bond1.1200
iface bond1.1200 inet manual
vlan_raw_device bond1

auto br1
iface br1 inet static
address 200.0.0.75
netmask 255.255.255.248
gateway 200.0.0.73
post-up echo 0 > /sys/devices/virtual/net/br1/bridge/multicast_snooping
bridge_ports bond1.1200
bridge_stp off

auto br1:1
iface br1:1 inet static
address 200.0.1.242
netmask 255.255.255.248
Mikhail Khirgiy
  • 2,073
  • 11
  • 7
  • This is a great possible solution, and Im eager to try it shortly. I am not necessarily trying to bridge the primary subnet ... those single IP addresses (public and private) Im trying to keep for the host. The other subnet would be bridged. – hmorton Mar 25 '17 at 21:19
  • A bridge interface is the same as virtual unmanaged switch. – Mikhail Khirgiy Mar 26 '17 at 03:54
  • One thing about this solution, according to the ISP these subnets are VLANs, does that make a difference to your solution above? Also, is there a way to reserve the primary IPs (public and private) just for the host machine? It is my understanding that if I bridge those addresses, they would be accessible to the VMs as well. – hmorton Mar 26 '17 at 17:26
  • Yes, vlan usage gives you needing to change the configuration file. Your server has got all ip addresses (not all networks) which was written in the interfaces configuration file. Please update your question to describe what vlan number (tag) has each network. Also add all networks, which clients want to have access to your server from and what vlan will be used for it. – Mikhail Khirgiy Mar 26 '17 at 18:14
  • I posted my VLAN information above - let me know what you think – hmorton Mar 27 '17 at 07:29
  • Ok. I see that you don't understand vlans. Is LAG ports on switch in access or trunk vlan mode? – Mikhail Khirgiy Mar 27 '17 at 15:19
  • According to ISP: VLAN's are setup as trunk mode when connected to a machine. – hmorton Mar 28 '17 at 12:40
  • I've updated my answer. Vlan 1200 is for public networks, vlan 1201 is for private networks. – Mikhail Khirgiy Mar 28 '17 at 14:46
  • OK I was finally able to attempt this. I am getting this error when I try to run it (Im only running the public VLANs section so that I still have access to the machine on the private net: – hmorton Mar 28 '17 at 19:51
  • `Mar 28 14:03:21 loro ifup[28154]: interface bond1.1200 does not exist! Mar 28 14:03:21 loro ifup[28154]: Waiting for br1 to get ready (MAXWAIT is 20 seconds). Mar 28 14:03:21 loro ifup[28154]: RTNETLINK answers: File exists Mar 28 14:03:21 loro ifup[28154]: Failed to bring up br1. Mar 28 14:03:21 loro ifup[28154]: RTNETLINK answers: File exists Mar 28 14:03:21 loro ifup[28154]: Failed to bring up br1:1.` – hmorton Mar 28 '17 at 19:52
  • To maintain privacy my VLAN numbers and IP addresses are actually different than what I'm using here on serverfault, but I've confirmed that Im using the correct numbers and IPs in my actual configuration – hmorton Mar 28 '17 at 19:58
  • You need to install `apt install vconfig` – Mikhail Khirgiy Mar 28 '17 at 20:20
  • Also if you restart networking via `service` command you should stop all virtual machines. They don't allow shutdown bridge interface. – Mikhail Khirgiy Mar 29 '17 at 05:41
  • Hi Mikhail - thank you so much for your time and advice on this. I installed vlan (ubuntu) and tried your configuration for the external interface only (so I still get into the machine if a problem happens). Service restart worked without any errors, but I was unable to get any external route into or out of the host. At this point I may find someone I can hire to help out on the job, but I really appreciate your help. – hmorton Mar 30 '17 at 09:57
  • You must have only one gateway on your server as I wrote in my config. – Mikhail Khirgiy Mar 30 '17 at 15:15