Forward traffic between VLANs with iptables

Question

We're having the following 4 VLANs on my network, connected to an Ubuntu Linux box with DHCP. This Linux should act as the L3 router as well.

VLAN 10  on interface eth1.10  with subnet 10.10.10.0/24
VLAN 20  on interface eth1.20  with subnet 10.10.20.0/24
VLAN 50  on interface eth1.50  with subnet 10.10.50.0/24
VLAN 100 on interface eth1.100 with subnet 10.10.100.0/24

Here is the /etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 192.168.10.169
        netmask 255.255.255.0
        gateway 192.168.10.1

auto eth1
iface eth1 inet static
        address 10.10.0.1
        network 10.10.0.0
        netmask 255.255.255.0
        broadcast 10.10.0.255

auto eth1.100
iface eth1.100 inet static
        address 10.10.100.1
        network 10.10.100.0
        netmask 255.255.255.0
        broadcast 10.10.100.255

auto eth1.10
iface eth1.10 inet static
        address 10.10.10.1
        network 10.10.10.0
        netmask 255.255.255.0
        broadcast 10.10.10.255

auto eth1.20
iface eth1.20 inet static
        address 10.10.20.1
        network 10.10.20.0
        netmask 255.255.255.0
        broadcast 10.10.20.255

auto eth1.50
iface eth1.50 inet static
        address 10.10.50.1
        network 10.10.50.0
        netmask 255.255.255.0
        broadcast 10.10.50.255

Now, all clients from all VLANs should be able to connect to the public internet via the interface eth0. That part actually works with the iptables rule -A POSTROUTING -o eth0 -j MASQUERADE. The DHCP server is working as well.

BUT since the VLAN 100 is going to be the network for administration computers, the clients in VLAN 100 should be able to access all other computers on the VLANs 10, 20 and 50. AND the clients in the VLANs 10, 20 and 50 should only be able to access computers inside their own VLANs.

I've tried so far the following iptables rules alongside with the MASQUERADE:

-A FORWARD -i eth1.100 -o eth1.10 -j ACCEPT
-A FORWARD -i eth1.100 -o eth1.20 -j ACCEPT
-A FORWARD -i eth1.100 -o eth1.50 -j ACCEPT
-A FORWARD -i eth1.10 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1.20 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1.50 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT

I tried to ping a PC with IP-address 10.10.20.100 from a PC with IP-address 10.10.100.101, but it didn't succeed. Nor was I able to ping 10.10.50.101 from the 100-subnet.

AND there is a weird side-behavior: The VLAN 20 is by accident (I have no idea why) acting like the 100 should. From there I can ping PCs in the VLANs 10 and 100, which should not be possible in the end.

I have IPv4 forwarding enabled in the kernel and the external internet works as expected.

Here is the complete output of iptables-save:

# Generated by iptables-save v1.6.0 on Thu May 26 09:28:59 2016
*filter
:INPUT ACCEPT [7375:724156]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [3695:415474]
-A FORWARD -i eth1.100 -o eth1.10 -j ACCEPT
-A FORWARD -i eth1.100 -o eth1.20 -j ACCEPT
-A FORWARD -i eth1.100 -o eth1.50 -j ACCEPT
-A FORWARD -i eth1.10 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1.20 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1.50 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Thu May 26 09:28:59 2016
# Generated by iptables-save v1.6.0 on Thu May 26 09:28:59 2016
*nat
:PREROUTING ACCEPT [32796:9980970]
:INPUT ACCEPT [142:30526]
:OUTPUT ACCEPT [1829:211124]
:POSTROUTING ACCEPT [128:29756]
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Thu May 26 09:28:59 2016

And the output of route:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.10.1    0.0.0.0         UG    0      0        0 eth0
10.10.0.0       *               255.255.255.0   U     0      0        0 eth1
10.10.10.0      *               255.255.255.0   U     0      0        0 eth1.10
10.10.20.0      *               255.255.255.0   U     0      0        0 eth1.20
10.10.50.0      *               255.255.255.0   U     0      0        0 eth1.50
10.10.100.0     *               255.255.255.0   U     0      0        0 eth1.100
link-local      *               255.255.0.0     U     1000   0        0 eth0
192.168.10.0    *               255.255.255.0   U     0      0        0 eth0

Thanks in advance, hope someone will figure out which iptables rules should I use to achieve what I want (if that's the problem at all).

EDIT:

As @Sanael requested, I did some more logging (-A FORWARD -o eth1+ -p icmp -j LOG --log-prefix "IPTABLES FORWARD: " --log-level 7) and here are the results:

Ping 10.10.20.101 --> 10.10.50.100 succeeded with the following log:

May 26 12:14:57 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.50 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.50.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=23708 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=96
May 26 12:14:57 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:3c SRC=10.10.50.100 DST=10.10.20.101 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15185 PROTO=ICMP TYPE=0 CODE=0 ID=1 SEQ=96
May 26 12:14:58 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.50 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.50.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=23709 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=97
May 26 12:14:58 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:3c SRC=10.10.50.100 DST=10.10.20.101 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15317 PROTO=ICMP TYPE=0 CODE=0 ID=1 SEQ=97
May 26 12:14:59 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.50 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.50.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=23710 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=98
May 26 12:14:59 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:3c SRC=10.10.50.100 DST=10.10.20.101 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15349 PROTO=ICMP TYPE=0 CODE=0 ID=1 SEQ=98
May 26 12:15:00 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.50 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.50.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=23711 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=99
May 26 12:15:00 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:3c SRC=10.10.50.100 DST=10.10.20.101 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15597 PROTO=ICMP TYPE=0 CODE=0 ID=1 SEQ=99

Ping 10.10.20.101 --> 10.10.100.100 failed with the following log:

May 26 12:09:06 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.100.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=18715 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=85
May 26 12:09:11 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.100.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=18716 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=86
May 26 12:09:16 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.100.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=18717 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=87

Ping 10.10.50.100 --> 10.10.20.101 failed with the following log:

May 26 12:11:28 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.20.101 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=54378 DF PROTO=ICMP TYPE=8 CODE=0 ID=1903 SEQ=1
May 26 12:11:29 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.20.101 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=54495 DF PROTO=ICMP TYPE=8 CODE=0 ID=1903 SEQ=2
May 26 12:11:30 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.20.101 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=54693 DF PROTO=ICMP TYPE=8 CODE=0 ID=1903 SEQ=3

Ping 10.10.50.100 --> 10.10.100.100 failed with the following log:

May 26 12:12:24 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.100.100 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=8069 DF PROTO=ICMP TYPE=8 CODE=0 ID=1905 SEQ=1
May 26 12:12:25 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.100.100 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=8269 DF PROTO=ICMP TYPE=8 CODE=0 ID=1905 SEQ=2
May 26 12:12:26 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.100.100 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=8375 DF PROTO=ICMP TYPE=8 CODE=0 ID=1905 SEQ=3

Ping 10.10.100.100 --> 10.10.20.101 failed BUT didn't create any log output.

Ping 10.10.100.100 --> 10.10.50.100 succeeded BUT didn't create any log output.

It looks ok. Could you please log the packets with `-A FORWARD -o eth1+ -p icmp -j LOG --log-prefix "IPTABLES FORWARD: " --log-level 7` and add to your question the logs of the 100 to 20 ping, and the ones from 20 to anything else ? You can get them with `journalctl -xef -n 50 -k`. — setenforce 1, May 26 '16 at 08:38
Thanks for the suggestion @Sanael I did now the logging and please see my updated answer for the log data. — AirPett, May 26 '16 at 09:37
is there a reason you can't do this on your switch, do all of these VLANS terminate on the Linux box? - I would use openvswitch — Sum1sAdmin, May 26 '16 at 09:44
The switch is a Cisco Catalyst 2950G, which is a Layer 2 device. It can't do cross-VLAN routing since it's not a L3 router. And what do you mean with the terminating? All VLANs go to the public internet through this Linux box, which is connected to the switch by an 802.1Q VLAN trunk. — AirPett, May 26 '16 at 09:52
I would terminate them on an L3 device such as firewall router or openvswitch - that would be my preference for ease of admin, can you add virtual interfaces to your firewall? - just put the VLAN gateways there no? — Sum1sAdmin, May 26 '16 at 09:57
Ok, I don't have a physical L3 device, but I could explore openvswitch. Currently my firewall is iptables(netfilter) and the VLANs are virtual interfaces for the eth1 (i.e. eth1.20, eth1.50). And do you mean the VLAN trunk with the VLAN gateway? — AirPett, May 26 '16 at 10:06
well, the same as your current setup really, any host on those vlans will have a DF of you linux box via trunk, and you have out of band management on 192.168.10.0/24 - you are essentially building a firewall router the manual way - but wont get VPN etc,etc. I would have a look at some of these http://www.mondaiji.com/blog/other/it/10175-the-hunt-for-the-ultimate-free-open-source-firewall-distro — Sum1sAdmin, May 26 '16 at 10:17
One thing, if there is going to be a lot of inter-vlan traffic you will be doing it all at the edge of your network and this could slow down internet access for everyone. — Sum1sAdmin, May 26 '16 at 10:19
Ok, I'll check OVS, but the only inter-vlan traffic is going to be ssh connections from the 100-vlan to the others to maintain servers in those networks. That's the reason why I'd really like not to teach myself a completely new system just now, and it should be possible to do this little thing with netfilter/iptables as well. — AirPett, May 26 '16 at 10:24
One more thing, If you take a look at the routing table in my question, should the other routes be having a gateway as well? like 10.10.10.0/24 gateway is 10.10.10.1. — AirPett, May 26 '16 at 10:31
no just hosts should have their gateways as the interface on the Linux box, they go to the linux interface for anything outside of their VLAN. something like this http://hitecksqud.blogspot.ie/2012/12/intervlan-routing-using-centos-with-1.html — Sum1sAdmin, May 26 '16 at 10:48

score 3 · Answer 1 · answered May 26 '16 at 12:20

First, your default policy in the FORWARD chain is ACCEPT, so you're not actually denying any traffic. This explains why the 20 -> 50 forwarding works. You can fix this with iptables -P FORWARD DROP.

As far as the other pings, notice that in your log entries, the TTL is either 127 or 63, indicating the firewall has already made its routing decision and decremented the TTL. Also, your firewall is not configured to block any traffic. This suggests the problem is in the routing configuration of the clients. May I suggest you use a sniffer on the ping recipient to see if the original ping packet is arriving? Make sure the default gateway for clients (not the firewall) on 10.10.10.0/24 is 10.10.10.1, etc. Good luck!

And following up on the routing table comment, your routing table is correct, indicating that your firewall is directly connected to your VLANed networks.

Thanks for the answer, I now changed the default policy to `DROP` and that part of the problem is now solved. I also figured out the big problem, please see my own answer in a monent. — AirPett, May 26 '16 at 13:02

score 2 · Accepted Answer · answered May 26 '16 at 13:12

The problem is now at least partly solved (I'm fine with this).

I was testing the network with Windows clients in the beginning, and both PCs had got the Windows 10 upgrade lately. The problem itself was really stupid: Windows didn't response to pings since there is some new security enabled by default in Windows 10. When I added one Linux laptop to the network I got to a situation where all others could ping the Linux laptop (default policy was ALLOW) but the Linux PC couldn't ping anything else (Windows didn't response to ping). Then I tried remote desktop to the Windows PCs in VLANs 10 and 50, from the Linux PC in VLAN 100 and boom - it worked!

So there was no problem at all with the iptables/netfilter rules.

Thank you very much for all answers and comments!

Here is my final, working and simplified iptables config:

# Generated by iptables-save v1.6.0 on Thu May 26 16:00:55 2016
*filter
:INPUT ACCEPT [359:39449]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [563:89333]
-A FORWARD -i eth1+ -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1.100 -o eth1+ -j ACCEPT
-A FORWARD -i eth1+ -o eth0 -j ACCEPT
-A FORWARD -i eth0 -o eth1+ -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Thu May 26 16:00:55 2016
# Generated by iptables-save v1.6.0 on Thu May 26 16:00:55 2016
*nat
:PREROUTING ACCEPT [5650:1147271]
:INPUT ACCEPT [91:14019]
:OUTPUT ACCEPT [325:31088]
:POSTROUTING ACCEPT [44:7161]
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Thu May 26 16:00:55 2016

score 0 · Answer 3 · answered May 26 '16 at 13:51

I. iptables

Did you add the log line after your existing rules ? If you did, there is what I think about your issue.

-j ACCEPT end the path of the packet in the rule. So a log line means your 6 rules didn't match.
The FORWARD default policies is ACCEPT, so packets are accepted anyway when they hit the end of the FORWARD chain. You should change this beahvior by adding iptables -P FORWARD DROP.
Seeing logs means the packet would be drop if you had the correct default policies. The 4 pings logged are from non-100 vlan, which is the expected behavior.
Seeing no log means the packet matched a rule, and was accepted. The two pings are from vlan 100, and matched a rule, which is once again the expected behavior.

We can see iptables is doing its job, you just need to change the default policy.

II. Routing

Now you need to understand where does the issue comes from. It could be the routing for example. Since the FORWARD default policy is ACCEPT (you can keep it until you found out the root cause of your problem), every single ping should have succeeded. Let's look at the ones who doesn't :
20 to 100, 50 to 20, 50 to 100, 100 to 20.

In the logs, we can check the MAC to know if the routing is correct. The format is <MAC dest (6 bytes)><MAC source (6 bytes)><08:00 & sometimes IP header>.
The logs we have are before the routing, so we have <eth1 mac><sender mac><not interesting>.
What we want is <receiver mac><eth1 mac><not interesting>, so we could see if the packet is send to the right place.
In order to do that, we could set the log rule in the POSTROUTING chain:

-A POSTROUTING -o eth1+ -p icmp -j LOG --log-prefix "IPTABLES POSTROUTING: " --log-level 7

What we know at the moment is :

10.10.20.101: c8:0a:a9:e5:f0:bc
10.10.50.100: 00:1e:ec:fa:d1:10
eth1: 00:0a:5e:50:7c:c1

We need to know the MAC for 10.10.10.100 too.
By the way, if you only have 4 vlans, try the 12 pings you can do to provide enough informations to debug.
Can you try the 8 ping from (and to) router interfaces? (eg. on the router: ping -I eth1.100 10.10.100.100)

III. Third party causes

Double check your Cisco conf (and open another question if you are almost sure the whitch is the root cause), verify every single vlan have internet access.

Edit: I'm writing way too slowly. Well done!

Thanks a lot for this excellent answer! You actually made me think of a possible problem: I can ping all clients in all VLANs from the Linux box running this all. From there all PCs (the Windows ones as well) respond, but when pinging from another subnet (VLAN 100), only Linux responds to ping even though I'm able to connect to the Win10 PCs with remote desktop from the same 100-VLAN. — AirPett, May 26 '16 at 17:06

Forward traffic between VLANs with iptables

EDIT:

3 Answers3

I. iptables

II. Routing

III. Third party causes