0

i'm trying to setup a linux machine as router for the following subnets:

                             +--------------------------------------------+                 host3
                             |                  router                    |                 10.1.0.254
                             |                                            |                    |
                             |                                            |                    |
                             |                                            |                    |
-----------------------------+ ---+  eth1                      eth3  +--- +--------------------+-----
                10.1.0.0/16  |    |  10.1.0.254            10.1.0.1  |    | 10.1.0.0/16
                subnet1      |    |                                  |    | subnet3
                             |    |                                  |    |
                             |    | forward                  forward |    |
                             |    |                                  |    |
                             |    |                                  |    |
-----------------------------+ ---+  eth2                      eth4  +--- +--------------------------
                10.2.0.0./16 |       10.2.0.254          10.2.0.254       | 10.2.0.0/16    
                subnet2      |                                            | subnet4        
                             +--------------------------------------------+

the router should:

  • forward traffic between subnet1 and sebnet2.
  • forward traffic between subnet3 and setnet4.
  • keep subnets 1+2 isolated from subnets 3+4

i've implemented this setup using

  • ip route (4 route tables, 1 per nic)
  • ip rule (each nic is associated with a its own route table)

this works well with one exception:

  • traffic gets forwarded between subnets 1 and 2.
  • traffic gets forwarded between subnets 3 and 4.
  • the one exception is host3
    • it cannot send/receive traffic from subnet4
    • it cannot get an arp reply for 10.1.0.1.
    • its ip is identical to the router's own ip on eth1.
    • if i give host3 a different ip on subnet3, all issues are resolved.
      for reasons outside the scope of this post i cannot change its ip.
      i also can't change eth1's ip.

googling similar issues pointed me to the local route table.

  • the local route table has a route entry for eth1's ip (created automatically)
  • this table has the highest priority in the system and appears to be hijacking host3's traffic.
  • i tried deleting this entry -
    • ip route del table local 10.1.0.254 dev eth1 proto kernel scope host src 10.1.0.254
    • it solves the issue, host3 can send/receives traffic from subnet4.
    • it's not a real solution however because eth1 stops replying to arp requests from subnet1.
      (unnoticeable as long as subnet1 hosts have its MAC cached).

what is the proper solution to this issue?
how do i keep the local route table from "coupling" these L3 domains i'm trying to keep separate?
how do i disable local route entries without loosing arp functionality?
(eth1-4 are only used for forwarding, they're never themselves the target of any tcp traffic)

Tomer
  • 13
  • 7
  • 1
    Routers route _between_ networks, not from a network back to the same network. You have identical networks on both sides of the "router". What you want to do is bridge, not route, between the identical networks. – Ron Maupin Nov 27 '17 at 23:50
  • you misread the question - the subnets that have identical CIDR are meant to be completely isolated from one another. – Tomer Nov 28 '17 at 09:05
  • i found that `echo 1 >> /proc/sys/net/ipv4/conf/all/accept_local` also resolved my issue – Tomer Dec 14 '17 at 16:45

2 Answers2

0

You could use network namespaces to separate your nics. One for the left side and one for the right side of your router. That way you don't need any additional routing table separation.

An introduction about network namespaces: https://blogs.igalia.com/dpino/2016/04/10/network-namespaces/

Hint: most articles about network namespaces put veth interfaces into the ns. But you can put any type of network interface into a namespace, like physical interfaces in your case.

Enno Gröper
  • 292
  • 1
  • 3
0

Check following scheme:

  1. One routing table for every separate routing domain (your router can forward packets inside routing domain, not between routing domains). Local routes allow to communicate with router itself from these domains.
#domain 1 (eth1 + eth2)
ip route add 10.1.0.0/16 dev eth1 table dom1
ip route add 10.2.0.0/16 dev eth2 table dom1
ip route add local 10.1.0.254 dev eth1 table dom1
ip route add local 10.2.0.254 dev eth2 table dom1

#domain 2 (eth3 + eth4)
ip route add 10.1.0.0/16 dev eth3 table dom2
ip route add 10.2.0.0/16 dev eth4 table dom2
ip route add local 10.1.0.1 dev eth3 table dom2
ip route add local 10.2.0.254 dev eth4 table dom2
  1. Create routing rules to separate the routing domains:
#domain 1
ip rule add iif eth1 lookup dom1 pref 101
ip rule add iif eth2 lookup dom1 pref 102
ip rule add oif eth1 lookup dom1 pref 103
ip rule add oif eth2 lookup dom1 pref 104

#domain 2
ip rule add iif eth3 lookup dom2 pref 201
ip rule add iif eth4 lookup dom2 pref 202
ip rule add oif eth3 lookup dom2 pref 203
ip rule add oif eth4 lookup dom2 pref 204
  1. Move the routing rule with 0 preference below your rules:
ip rule add from all lookup local pref 1000
ip rule del pref 0

There is some issue caused by duplication of address on eth2 and eth4 interfaces. You can solve it with iptables rules with CONNMARK.

Anton Danilov
  • 5,082
  • 2
  • 13
  • 23
  • i tried something very similar before posting this question, pointed me to my mistake - – Tomer Nov 30 '17 at 10:48
  • i tried something very similar before posting this question, this answer pointed me to my mistake - i was using `iptables --in-interface ... --set-mark ...` + `ip rule add fwmark ... table ...`. however since arp is not ip packets, these rules weren't enough to get arp working, `ip rule iif ... ` covers arp traffic. – Tomer Nov 30 '17 at 11:00