I have fairly regular LXD setup (version 3.0.3) with the default bridge interface lxdbr0 on Ubuntu 18.04.2.
On the other hand my host machines network is not the most regular. I have two bridges (br0 and br1), connected via two bonds to four interfaces (2 interfaces per bond, 1 bond per bridge).
Because I had hard time getting regular routing working properly I am using separate routing tables per bridge. I needed to get the requests and replies use the same egress path as ingress.
Now the issue is that I can ping everything and everywhere (host does IP masquerade) except hosts that are in the same sub net as br0 and br1 and as my forward proxy is in the same sub net as br1 I have problem. Host default route is also on br1.
The rules and routes for host machine are:
user@host:~$ ip route show
default via 10.10.50.254 dev br1 proto static
10.146.233.0/24 dev lxdbr0 proto kernel scope link src 10.146.233.1
10.10.99.0/24 dev br0 proto kernel scope link src 10.10.99.1
10.10.50.0/24 dev br1 proto kernel scope link src 10.10.50.1
user@host:~$ ip route show table 99
default via 10.10.99.254 dev br0 proto static
user@host:~$ ip rule show table 99
0: from 10.10.99.0/24 lookup 99
user@host:~$ ip route show table 50
default via 10.10.50.254 dev br1 proto static
user@host:~$ ip rule show table 50
0: from 10.10.50.0/24 lookup 50
The relevant netplan config as follows:
...
bridges:
br0:
addresses: [10.10.99.1/24]
interfaces: [bond1]
routes:
- to: 0.0.0.0/0
via: 10.10.99.254
table: 99
routing-policy:
- from: 10.10.99.0/24
table: 99
parameters:
stp: true
forward-delay: 0
br1:
addresses: [10.10.50.1/24]
gateway4: 10.10.50.254
interfaces: [bond0]
routes:
- to: 0.0.0.0/0
via: 10.10.50.254
table: 50
routing-policy:
- from: 10.10.50.0/24
table: 50
parameters:
stp: true
forward-delay: 0
The only IPTABLES rules are generated by LXD
user@host:~$ sudo iptables -L -nv
Chain INPUT (policy ACCEPT 828 packets, 62176 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- lxdbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 /* generated for LXD network lxdbr0 */
16 1064 ACCEPT udp -- lxdbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 /* generated for LXD network lxdbr0 */
2 658 ACCEPT udp -- lxdbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67 /* generated for LXD network lxdbr0 */
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
7 588 ACCEPT all -- * lxdbr0 0.0.0.0/0 0.0.0.0/0 /* generated for LXD network lxdbr0 */
7 588 ACCEPT all -- lxdbr0 * 0.0.0.0/0 0.0.0.0/0 /* generated for LXD network lxdbr0 */
Chain OUTPUT (policy ACCEPT 540 packets, 69728 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- * lxdbr0 0.0.0.0/0 0.0.0.0/0 tcp spt:53 /* generated for LXD network lxdbr0 */
16 1020 ACCEPT udp -- * lxdbr0 0.0.0.0/0 0.0.0.0/0 udp spt:53 /* generated for LXD network lxdbr0 */
2 680 ACCEPT udp -- * lxdbr0 0.0.0.0/0 0.0.0.0/0 udp spt:67 /* generated for LXD network lxdbr0 */
NAT rules are
user@host:~$ sudo iptables -t nat -L -nv
Chain PREROUTING (policy ACCEPT 1649 packets, 66895 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 9 packets, 899 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 10 packets, 927 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 10 packets, 927 bytes)
pkts bytes target prot opt in out source destination
3 252 MASQUERADE all -- * * 10.146.233.0/24 !10.146.233.0/24 /* generated for LXD network lxdbr0 */
if I ping non br0 or br1 subnet addresses I can see requests and replies on br1, lxdbr0 and veth interfaces. If I ping br0 or br1 addresses I see only requests on lxdbr0 and veth interfaces. No traffic on br1 nor replies on any of the before mentioned interfaces.
and finally my interfaces configuration with omitted eth, lo and bond interfaces:
user@host:~$ ip a
7: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ee:f1:6a:84:6f:df brd ff:ff:ff:ff:ff:ff
inet 10.10.50.1/24 brd 10.10.50.255 scope global br1
valid_lft forever preferred_lft forever
8: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:13:19:08:56:ad brd ff:ff:ff:ff:ff:ff
inet 10.10.99.1/24 brd 10.10.99.255 scope global br0
valid_lft forever preferred_lft forever
22: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether fe:cb:12:0f:9f:d7 brd ff:ff:ff:ff:ff:ff
inet 10.146.233.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fe80::70f5:3eff:feac:2ae6/64 scope link
valid_lft forever preferred_lft forever
26: vethANRBKX@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether fe:cb:12:0f:9f:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::fccb:12ff:fe0f:9fd7/64 scope link
valid_lft forever preferred_lft forever
My desired outcome would be to restrict NAT on 10.10.50.254 and 10.10.99.254 and access internet from the containers through my proxy in br1 sub network. I probably need to do something with my routing/policy rules or iptables or configure the lxdbr0 differently, but I seem to be stuck at the moment.
