Properly configure two eth interfaces on different subnets

Question

Can anyone with heavy background in both Linux and Networking help me out? I've scoured the Internet looking for a clear answer for this and I've even compared a box I have access to with the same setup and I can't get my box working.

The basic situation is this:

===========================

A RHEL 6 Linux box has two Ethernet interfaces:

eth0 = 10.10.5.10/24
eth1 = 10.10.6.10/24

[root@box ~]# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.10.5.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.10.6.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
0.0.0.0        10.10.5.254     0.0.0.0         UG        0 0          0 eth0

ip forwarding has been enabled and is active.

The problem:

Clients on the same subnet as eth1 or the 10.10.6.x network can ping and telnet to resources running on eth1 on this box.

However, clients on any other subnet cannot. So from my workstation I can ping and telnet to resources on the box's eth0 all day long, but nothing on eth1... What gives?

Comparing this setup to another box of a similar nature in my environment shows no discernible differences.

Here are some files:

# cat ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="none"
HWADDR="Omitted"
IPADDR="10.10.5.10"
NETMASK="255.255.255.0"
NETWORK="10.10.5.0"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="no"

# cat ifcfg-eth1
DEVICE="eth1"
BOOTPROTO="none"
HWADDR="omitted"
IPADDR="10.10.6.10"
NETMASK="255.255.255.0"
NETWORK="10.10.6.0"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="no"

# cat route-eth0 
default via 10.10.5.254 dev eth0

# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=box
GATEWAY=10.10.5.254
NOZEROCONF=yes

Any assistance you can provide will be appreciated!

Thanks.

Adding more info, but with brevity included:

# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    inet 10.10.5.10/24 brd 10.10.5.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    inet 10.10.6.10/24 brd 10.10.6.255 scope global eth1

is your eth0 interface up? can you reach anything on the `10.10.5.0/24` network? Can you reach your default gateway `10.10.5.254`? if you can't for either these, then you won't be able to get to anything outside of your `10.10.6.0/24` network, as your server doesn't know how to get outside of its local network. — emynd, Jun 06 '13 at 15:20
Yes, anything on the 5 network is reachable from the same subnet or a different subnet. — beaconfield, Jun 06 '13 at 15:22
i am specifically talking about from your server. Can your server reach its default gateway? — emynd, Jun 06 '13 at 15:24
After running a tcpdump I can see packets hitting the eth1 interface, so it looks like the packets are having an issue returning? Shouldn't they use the default gateway? And if they are, should they be using a different gateway somehow? — beaconfield, Jun 06 '13 at 15:24
@emynd Yes it can reach its default gateway. It has connectivity to its network on the switch and the larger network within the infrastructure. No problems there. — beaconfield, Jun 06 '13 at 15:24
@emynd `# ping 10.10.5.254 PING 10.10.5.254 (10.10.5.254) 56(84) bytes of data. 64 bytes from 10.10.5.254: icmp_seq=1 ttl=255 time=0.774 ms` — beaconfield, Jun 06 '13 at 15:26
`# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.10.5.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 10.10.6.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 0.0.0.0 10.10.5.254 0.0.0.0 UG 0 0 0 eth0` — beaconfield, Jun 06 '13 at 15:30
For brevity: `eth0 Link encap:Ethernet HWaddr omitted inet addr:10.10.5.10 Bcast:10.10.5.255 Mask:255.255.255.0` — beaconfield, Jun 06 '13 at 15:33
For brevity: `eth1 Link encap:Ethernet HWaddr omitted inet addr:10.10.6.10 Bcast:10.10.6.255 Mask:255.255.255.0` — beaconfield, Jun 06 '13 at 15:33
when you say "clients on any other subnet cannot" : which subnet are you talking about? 10.10.5.0 (coming from eth0-side switches) ? or another subnet (or coming on eth1?) — Olivier Dulac, Jun 06 '13 at 15:47
and please check that : `cat /proc/sys/net/ipv4/ip_forward` is set to `1` — Olivier Dulac, Jun 06 '13 at 15:49
@beaconfield `After running a tcpdump I can see packets hitting the eth1 interface, so it looks like the packets are having an issue returning? Shouldn't they use the default gateway? And if they are, should they be using a different gateway somehow?` This is because you are running `tcpdump` only on `eth1`. The packets will go out `eth0`, as that's where the default gateway is. — emynd, Jun 06 '13 at 15:55
@beaconfield: you need to have switches/hub with the 10.0.5.0/24 machines on one side, and have eth0 plugged on that side, and 10.0.6.0/24 on another side, and have eth1 plugged on that other side. Mixing the two would be both hairy and troublesome (subnet are there to separate networks)... is it the case? — Olivier Dulac, Jun 06 '13 at 16:38
@olivierDulac 1) I mean clients as is in any host with an ip on any network other than eth1's subnet that is NOT on the box itself. So from my workstation, I have access to both subnets, but I can only connect to the .5 subnet that eth0 runs on. 2) ip forwards is enabled and active — beaconfield, Jun 06 '13 at 17:13
@olivierDulac The box in question is a vSphere VM and NIC1 is attached to the .5 network while NIC2 is attached to the .6 network. — beaconfield, Jun 06 '13 at 17:14

Olivier Dulac · Answer 1 · 2013-06-06T18:19:12.257

with the setup as you currently show:

you have 2 sides :
- the eth1 side (10.10.6.10 interface, on 10.10.6.0/24 lan)
- the eth0 side (10.10.5.10 interface, on 10.10.5.0/24 lan)
- you defined that machine's default gateway on eth0 side (default gw ip: 10.10.5.254)
therefore
- on eth1 side:
  - [requisite] any machines in 10.10.6.0/24 should be able to communicate with 10.10.6.10 (and with each other as well) (I assume no firewalling issues)
  - [requisite] in addition: any machine in 10.10.6.0/24 THAT HAS defined their default gateway as "10.10.6.10" will be able to send packets to other machines, going through your machine (10.10.6.10/24->eth1[machine]eth0-> other lans). Actually you don't need to have it as default gateway, but for ANY lan on the eth0 side, they need to associate that LAN with the 10.10.6.10 gateway.
- on eth0 side:
  - [requisite] any machine on 10.10.5.0/24 should see 10.10.5.10 (eth0) and can communicate with it (and with each other).
  - [requisite] any machine on another LAN should be able to communicate with eth0, via (with maybe additionnal hops) the machine which has 10.10.5.254 ip (ex: if that machine have 2 subnets, 10.10.5.254/24 on the same side as eth0, and 10.x.y.z/24 on another interface, machine in 10.x.y.0/24 can send packets to 10.10.5.10 by having a route: 10.10.5.0/24 with gateway 10.x.y.z).
  - if so, then they just need to have an additionnal route to reach 10.10.6.0/24 : 10.10.6.0/24 via gateway 10.x.y.z

it can get complicated to put in words...

what would help: tell us the list of networks ( ) and each network's getways.

right now I guessed you try to have:

 [everything other machines (even internet, via the exterior internet gateway)]
                  |
                  |(links via switch(s) for 10.x.y.0/24 network, for example.
                  |   If you have such an additionnal LAN.
                  |   Maybe it's even "the Internet", and therefore not in 10.x anything,  
                  |   but this is just an example.)
                  |
                  |      
      (10.x.y.z?) |  [---------------]
   (or Internet?)ethZ[some router/box]ethW(10.10.5.254)
                     [---------------] |
                                       |
                                       |(links via switch(s) for 10.10.5.0/24 network)
                                       |
                             [-------] |
             (10.10.6.10)eth1[yourbox]eth0(10.10.5.10)
                          |  [-------]   
                          |
                          |(links via switch(s) for 10.10.5.0/24 network)
                          |
                          |  [---------------------------]
                         ethx[any machine in 10.10.6.0/24]
                             [---------------------------]

General way to debug your setup:

imagine you ARE the IP packet: you have a source IP, a destination IP.
Now, on the interface you sit on at the source, see what that machine's routes tells you which next hop is. And go there(but first ensure that machine's setup allows you to, for example on the eth0/eth1 machine, it needs ipforward to go from one interface to the other). And proceed, one hop at a time, toward your goal (AND back to your source : sometimes the destination machine's route/defaultgw are broken and therefore the reply packet is not sent via the right path)
- ex1 (very simple): 10.10.6.10 trying to send to 10.10.5.10: step1 you are on eth1 (10.10.6.10). That machine has a route saying "10.10.5.0/24 is directly here, on eth0". Ipforwarding allows you to hop from eth1 to eth0. Proceed to eth0. step2: you are now (same IP packet) on eth0. It turns out this is your destination: GOOD. step 2b: reply packet: inverse source/destination : you are now a packet (10.10.5.10 -> 10.10.6.10=): thank god, a local route tells you the next hop is on eth1. step4: no eth1 : you are at destination. GOOD.
- ex2 (a bit longer ex:) you are a packet from "any_machine", 10.10.6.12 (on the eth1 side). you try to reach as destinatino google (8.8.8.8, for example). So you are "10.10.6.12 -> 8.8.8.8" and at first you sit on the 10.10.6.12 interface (say, eth3). A local route on "any_machine" tells you (as it was the prerequisite, above): "default gw=10.10.6.10", so you hop to "yourbox", and arrive there on the eth1 side. step2: you are now on "yourbox/eth1" : now local routes tell you "you need to go to default gw: 10.10.5.254, via eth0" (and ipforward on "yourbox" allows you to hop from eth1 to eth0). Now you arrive on eth0, and following the default gw route, you arrive later on "somerouter" on ethW(10.10.5.254). etc... (and don't forget to check the way-back route)

thank you for your response. I'm still looking it over, but: if I `ping -I eth1 10.10.6.254` it works. If I `ping -I eth1 10.10.5.254` it does not work. The reverse is true of pinging from eth0. It can ping its default gateway, but cannot ping eth1's default gateway of 10.10.6.254 on the switch. — beaconfield, Jun 06 '13 at 17:42
2 default gateways for your box ?? so if you try to reach, say, google, you send it both to eth1 to 10.10.6.254 AND to eth0 to 10.10.5.254 ??? — Olivier Dulac, Jun 06 '13 at 17:46
stick with 1 default gateway (that's its purpose: to be the gateway where everything NOT reachable directly via the Ethernet layer is sent to the (SINGLE) default gateway, UNLESS for some specific other LANs you define another special gateway to reach those... — Olivier Dulac, Jun 06 '13 at 17:47
Oh no, I'm sorry, I just defined the def. gateways for those subnets on the switch level. — beaconfield, Jun 06 '13 at 17:48
there is only one default gateway on the box and it is: `10.10.5.254` — beaconfield, Jun 06 '13 at 17:49
to debug your setup: imagine you ARE the IP packet: you have a source IP, a destination IP. Now, on the interface you sit on, see what routes will lead to to where (and ensure that machine's setup allows you to, for example on the eth0/eth1 machine, it needs ipforward to go from one interface to the other). And proceed, one hop at a time, toward your goal (AND back to your source : sometimes the destination machine's route/defaultgw are broken and therefore the reply packet is not sent via the right path) — Olivier Dulac, Jun 06 '13 at 17:49
let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/9133/discussion-between-beaconfield-and-olivier-dulac) — beaconfield, Jun 06 '13 at 17:51

score 1 · Answer 2 · answered Jun 06 '13 at 17:11

1

Your default gateway is at 10.10.5.254, however the eth1 interface is configured as 10.10.6.10/24. What this effectively means is that the 10.10.6.x interface cannot hit the default gateway as they are on different subnets.

More general network information is needed to properly assess your options, however your two options from what has been posted so far are to:

Confirm that the two networks are in fact /24 subnets, if so, set the proper gateway for eth1 in that interface's config file
If the interfaces are part of a larger supernet, configure the two interfaces to exist on a /16 subnet to use the same gateway.

answered Jun 06 '13 at 17:11

NcA

441
2
8

thanks for your reply - the subnets are in fact /24 netmasks. So what are you saying? Define a line in the ifcfg-eth1 that says: `GATEWAY=10.10.6.254` or are you saying I need to add some sort of static route? – beaconfield Jun 06 '13 at 17:18
Exactly that, as long as 10.10.6.254 is indeed your gateway for that /24. It's preferred to do that for the other interface/network too. Further routing rules may need to be added to define the default routes for the system as a whole. – NcA Jun 06 '13 at 17:21
@NcA : you probably confuse his box (having 2 interfaces, with the default gw on eth0) with a regular box with only 1 interface (therefore needing routes and/or a default gateway to reach other networks) : on his box, if ip_forward is 1, packets from eth1 can reach other lan's by going via eth0 (and its defautl gw)... IF you add a default gw on eth1 as well, packets by default will go through BOTH (and you better have a sane network behind, with each default gw knowing to avoid loops, to avoid hairy routing problems and superflous communications) – Olivier Dulac Jun 06 '13 at 18:15

score 0 · Answer 3 · answered Jun 06 '13 at 18:29

Have you enabled IP forwarding?

sysctl net.ipv4.ip_forward

If it says 0 you have not and your box is not routing packets.

Enable it by

sysctl -w net.ipv4.ip_forward=1

And make it permanent by adding the following to /etc/sysctl.conf:

net.ipv4.ip_forward = 1

Properly configure two eth interfaces on different subnets

3 Answers3

Linked