ip routing for extra eni on RHEL ec2 instances

Question

I have noticed that when one add an additional ENI on an ec2, getting routing to work is so difficult and there is no documentation anywhere to refer to.

I do know that if one uses the Amazon Linux ami, everything works well automatically without any extra work. But using any other ami, one has to configure the routing to work.

non-Amazon Linux

[root@ip-10-11-2-20 ~]# cat /etc/*release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.3 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.3"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.3 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.3:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.3"
Red Hat Enterprise Linux Server release 7.3 (Maipo)
Red Hat Enterprise Linux Server release 7.3 (Maipo)

[root@ip-10-11-2-20 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.11.2.1       0.0.0.0         UG    100    0        0 eth0
0.0.0.0         10.11.2.1       0.0.0.0         UG    101    0        0 eth1
10.11.2.0       0.0.0.0         255.255.255.0   U     100    0        0 eth0
10.11.2.0       0.0.0.0         255.255.255.0   U     101    0        0 eth1
[root@ip-10-11-2-20 ~]# ip route
default via 10.11.2.1 dev eth0  proto static  metric 100
default via 10.11.2.1 dev eth1  proto static  metric 101
10.11.2.0/24 dev eth0  proto kernel  scope link  src 10.11.2.20  metric 100
10.11.2.0/24 dev eth1  proto kernel  scope link  src 10.11.2.247  metric 101

[root@ip-10-11-2-20 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.11.2.20  netmask 255.255.255.0  broadcast 10.11.2.255
        inet6 fe80::1012:30ff:feca:4bd0  prefixlen 64  scopeid 0x20<link>
        ether 12:12:30:ca:4b:d0  txqueuelen 1000  (Ethernet)
        RX packets 3065  bytes 258041 (251.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2375  bytes 351169 (342.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.11.2.247  netmask 255.255.255.0  broadcast 10.11.2.255
        inet6 fe80::1034:1bff:fe8d:c742  prefixlen 64  scopeid 0x20<link>
        ether 12:34:1b:8d:c7:42  txqueuelen 1000  (Ethernet)
        RX packets 473  bytes 22008 (21.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 58  bytes 5544 (5.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 140  bytes 11924 (11.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 140  bytes 11924 (11.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@ip-10-11-2-20 ~]# ping -c 3 10.11.2.20
PING 10.11.2.20 (10.11.2.20) 56(84) bytes of data.
64 bytes from 10.11.2.20: icmp_seq=1 ttl=64 time=0.013 ms
64 bytes from 10.11.2.20: icmp_seq=2 ttl=64 time=0.027 ms
64 bytes from 10.11.2.20: icmp_seq=3 ttl=64 time=0.025 ms

--- 10.11.2.20 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.013/0.021/0.027/0.008 ms

[root@ip-10-11-2-20 ~]# ping -c 3 10.11.2.247
PING 10.11.2.247 (10.11.2.247) 56(84) bytes of data.
64 bytes from 10.11.2.247: icmp_seq=1 ttl=64 time=0.013 ms
64 bytes from 10.11.2.247: icmp_seq=2 ttl=64 time=0.024 ms
64 bytes from 10.11.2.247: icmp_seq=3 ttl=64 time=0.024 ms

--- 10.11.2.247 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.013/0.020/0.024/0.006 ms

Amazon Linux

[root@ip-10-11-2-149 ~]# cat  /etc/*release
NAME="Amazon Linux AMI"
VERSION="2017.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2017.03"
PRETTY_NAME="Amazon Linux AMI 2017.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2017.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2017.03

[root@ip-10-11-2-149 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.11.2.1       0.0.0.0         UG    0      0        0 eth0
0.0.0.0         10.11.2.1       0.0.0.0         UG    10001  0        0 eth1
10.11.2.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.11.2.0       0.0.0.0         255.255.255.0   U     0      0        0 eth1
169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0

[root@ip-10-11-2-149 ~]# ip route
default via 10.11.2.1 dev eth0
default via 10.11.2.1 dev eth1  metric 10001
10.11.2.0/24 dev eth0  proto kernel  scope link  src 10.11.2.149
10.11.2.0/24 dev eth1  proto kernel  scope link  src 10.11.2.61
169.254.169.254 dev eth0

[root@ip-10-11-2-149 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 12:B2:B8:77:D0:F8
          inet addr:10.11.2.149  Bcast:10.11.2.255  Mask:255.255.255.0
          inet6 addr: fe80::10b2:b8ff:fe77:d0f8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:1490 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1441 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:129285 (126.2 KiB)  TX bytes:143368 (140.0 KiB)

eth1      Link encap:Ethernet  HWaddr 12:9B:45:32:EB:BA
          inet addr:10.11.2.61  Bcast:10.11.2.255  Mask:255.255.255.0
          inet6 addr: fe80::109b:45ff:fe32:ebba/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:263 errors:0 dropped:0 overruns:0 frame:0
          TX packets:347 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:16825 (16.4 KiB)  TX bytes:25705 (25.1 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:140 (140.0 b)  TX bytes:140 (140.0 b)

[root@ip-10-11-2-149 ~]# ping -c 3 10.11.2.149
PING 10.11.2.149 (10.11.2.149) 56(84) bytes of data.
64 bytes from 10.11.2.149: icmp_seq=1 ttl=255 time=0.018 ms
64 bytes from 10.11.2.149: icmp_seq=2 ttl=255 time=0.027 ms
64 bytes from 10.11.2.149: icmp_seq=3 ttl=255 time=0.026 ms

--- 10.11.2.149 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2035ms
rtt min/avg/max/mdev = 0.018/0.023/0.027/0.006 ms

[root@ip-10-11-2-149 ~]# ping -c 3 10.11.2.61
PING 10.11.2.61 (10.11.2.61) 56(84) bytes of data.
64 bytes from 10.11.2.61: icmp_seq=1 ttl=255 time=0.018 ms
64 bytes from 10.11.2.61: icmp_seq=2 ttl=255 time=0.027 ms
64 bytes from 10.11.2.61: icmp_seq=3 ttl=255 time=0.029 ms

--- 10.11.2.61 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2034ms
rtt min/avg/max/mdev = 0.018/0.024/0.029/0.007 ms

Now here is the problem BELOW...see how the non-mazon Linux can ping both eth0 and eth1 ENIs on the Amazon Linux but the Amazon Linux cannot ping the eth1 of the non-Amazon Linux

INTER NETWORKING

non-Amazon Linux

[root@ip-10-11-2-20 ~]# ping -c 3 10.11.2.149
PING 10.11.2.149 (10.11.2.149) 56(84) bytes of data.
64 bytes from 10.11.2.149: icmp_seq=1 ttl=255 time=0.629 ms
64 bytes from 10.11.2.149: icmp_seq=2 ttl=255 time=0.486 ms
64 bytes from 10.11.2.149: icmp_seq=3 ttl=255 time=0.472 ms

--- 10.11.2.149 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.472/0.529/0.629/0.070 ms

[root@ip-10-11-2-20 ~]# ping -c 3 10.11.2.61
PING 10.11.2.61 (10.11.2.61) 56(84) bytes of data.
64 bytes from 10.11.2.61: icmp_seq=1 ttl=255 time=0.595 ms
64 bytes from 10.11.2.61: icmp_seq=2 ttl=255 time=0.560 ms
64 bytes from 10.11.2.61: icmp_seq=3 ttl=255 time=0.522 ms

--- 10.11.2.61 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.522/0.559/0.595/0.029 ms

Amazon Linux

[root@ip-10-11-2-149 ~]# ping -c 3 10.11.2.20
PING 10.11.2.20 (10.11.2.20) 56(84) bytes of data.
64 bytes from 10.11.2.20: icmp_seq=1 ttl=64 time=0.446 ms
64 bytes from 10.11.2.20: icmp_seq=2 ttl=64 time=0.465 ms
64 bytes from 10.11.2.20: icmp_seq=3 ttl=64 time=0.481 ms

--- 10.11.2.20 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2030ms
rtt min/avg/max/mdev = 0.446/0.464/0.481/0.014 ms

[root@ip-10-11-2-149 ~]# ping -c 3 10.11.2.247
PING 10.11.2.247 (10.11.2.247) 56(84) bytes of data.
^C
--- 10.11.2.247 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2028ms

How can i allow the routing to the eth1 extra ENI added to the RHEL 7 ec2 instance?

score 1 · Answer 1 · answered Aug 06 '17 at 23:28

Just to add exactly what i did and things started to work

Just added the following 4 lines from the URL @strongjz mentioned

non-Amazon Linux

ip route add default via 10.11.2.1 dev eth0 tab 1
ip route add default via 10.11.2.1 dev eth1 tab 2

ip rule add from 10.11.2.20/32 tab 1 priority 500
ip rule add from 10.11.2.247/32 tab 2 priority 600

And the rest was history..all ping works to all interfaces, both eth0 and eth1!!!

All thanks to this URL https://forums.aws.amazon.com/message.jspa?messageID=404398

score 0 · Accepted Answer · answered Aug 06 '17 at 18:27

I have had this problem before and found this blog post to be helpful

Something about the routes not being setup properly. I have added this to the Userdata script so when it boots the routes get added properly. It's assuming that a new ENI us being attached at eth1.

# Use the AWS CLI to get the id of the ENI to be attached
  NETWORK_INTERFACE_ID=`aws ec2 describe-network-interfaces --filters "Name=status,Values=available" "Name=tag:Name,Values=$ENI_NAME" "Name=subnet-id,Values=$SUBNET_ID" --output json --query "NetworkInterfaces[0].NetworkInterfaceId" | grep -o 'eni-[a-z0-9]*'`
  # Attach the ENI (and display the attachment id)
  echo "Attaching ENI:$ENI_NAME  NETID:$NETWORK_INTERFACE_ID Index:$ENI_INDEX"

  aws ec2 attach-network-interface --network-interface-id $NETWORK_INTERFACE_ID --instance-id $SELF_INSTANCE_ID --device-index $ENI_INDEX


  printf '\nauto eth1\niface eth1 inet dhcp' >> /etc/network/interfaces.d/eth1.cfg

  GATEWAY=`ip route | awk '/default/ { print $3 }'`
  #get the secondary ip
  SECOND_IP=`aws ec2 describe-network-interfaces --filters  "Name=tag:Name,Values=$ENI_NAME" "Name=subnet-id,Values=$SUBNET_ID" --output text --query "NetworkInterfaces[0].PrivateIpAddress"`
  #add the routes



  echo "Adding the IP routes and rules for secondary $SECOND_IP"
  printf "\nup ip route add default via $GATEWAY dev eth1 tab 2\n" >> /etc/network/interfaces.d/eth1.cfg
  printf "up ip rule add from $SECOND_IP/32 tab 2 prio 1000\n" >> /etc/network/interfaces.d/eth1.cfg

  sleep 20 # the attach takes some time to happen

  systemctl restart networking.service

  ifup eth1

  ip rule list

  ifconfig

yes i finally got it to work..few hours after posting the question from the same exact URL..that was a life saver..will accept your answer..was going to type the answer later..but since you answered with that URL..will accept..thanks — uberrebu, Aug 06 '17 at 23:22

ip routing for extra eni on RHEL ec2 instances

2 Answers2

Linked