Configuring multiple ENIs on an EC2 instance

Question

How do you configure the interfaces in the OS without using ec2-net-utils?

I need to configure around 18 public ips to be accessible on a single EC2 instance. Each public IP needs to be bound to a single internal IP on the instance. I have 2 ENIs configured with the necessary Elastic IPs and these are attached to the instance but I am stuck at getting the OS recognise the ENIs.

I have been following this tutorial but can't work out how to extend it to configure multiple IPs per interface. Also there seems to be some debate according to this question about what the correct way to do this is.

If anyone would be able to point me in the right direction I would be really grateful. I have nearly wasted 2 days on this!

Here are the details:

• OS: Centos 7
• ENI1 : Secondary Private IPs = [10.0.1.97 ...10.0.1.106]
• ENI2 : Secondary Private IPs = [10.0.1.218 ... 10.0.1.226]
• VPC gateway 10.0.1.1

Answer 1

Since the previous answer is clutter, I will put another one with workflow, just by using AWS CLI (you can write fine tune automation script using AWS SDK) (http://docs.aws.amazon.com/cli/latest/userguide/installing.html)

1. Create ENI with private IP using. (aws ec2 create-network-interface) Write down ENI id
2. Allocate EIP for VPC. (aws ec2 allocate-address --domain vpc) write down EIP-id
3. Link EIP to ENI-id, point to correct private IP (aws ec2 associate-address)
4. Create or launch EC2 instance, attach to the ENI. (aws ec2 attach-network-interface)

Once you put everything script in place, recreate the EC2 instance with the proper ENI is just matter of minutes.

(Updated answer below) In Linux, to assign multiple IP address to an interface, the correct assignment to the interface is to add additional ip address to the physical interface. For ubuntu , etc, it is something like eth0:0 , eth0:1, for 1st interface, eth1:0, eth1:1 for subsequent interface.

And this is slightly different for Centos, ie.

enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether ....
inet 192.168.1.150/24 brd 192.168.1.255 scope global enp0s3
inet 192.168.1.151/24 brd 192.168.1.255 scope global secondary enp0s3
inet 192.168.1.152/24 brd 192.168.1.255 scope global secondary enp0s3

So the correct documentation should be this one. http://www.unixmen.com/linux-basics-assign-multiple-ip-addresses-single-network-interface-card-centos-7/

In short, Centos will automatically create one network-interface file for each interface. Just go /etc/sysconfig/network-scripts/ and check each file name as ifcfg-eth* (don't ask me why the above link show enp0) . The tricky part is whether your Centos enforce to use NetworkManager, and you must configure as required by the centos documentation wiki.centos.org/FAQ/CentOS7

So you should see least 4 interface file for your m3.xlarge , e.g. /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 /etc/sysconfig/network-scripts/ifcfg-eth2 /etc/sysconfig/network-scripts/ifcfg-eth3

Since the main interface should be running, it will give you the hint of the running instance IP address. So if you open /etc/sysconfig/network-scripts/ifcfg-eth0, if the instance configure as IP 10.0.1.10, you should see this

IPADDR0="10.0.1.10"

To add more IP address, just do as unixmen link say, i.e.

IPADDR1="10.0.1.97"
IPADDR2="10.0.1.98"
IPADDR3="10.0.1.99"

Then open ifcfg-eth1, ifcfg-eth2, ifcfg-eth3 and do repeat the task. After done that, use "systemctl restart network" to restart. (backup all config file so you just override the file in the future)

Next , you play with the routing part. Just print out your VPC route table, Subnet. Now inside /etc/sysconfig/network-scripts/, you deal with route-ethX , i.e. route-eth0, route-eth1, route-eth2,route-eth3. That's mean, you must know your own network to configure this part (which should be shown in your VPC layout). Since you mentioned only one VPC gateway 10.0.1.1, so for all route-* file, it should be something like this

# I just assume your put all your 10.0.1.x in the CIDR /24 segments
#
# file route-eth0
# Assume your first ENI IP address is  10.0.1.10 
# format : default via gateway-ip dev dev-name table route-table-number 
default via 10.0.1.1 dev eth0 table 0

# format : network cidr  dev dev-name src ENI-intrace-IP route-table-number  
10.0.1.0/24 dev eth0 src 10.0.1.10 table 0

# file route-eth1
# Assume your 2nd ENI IP address is  10.0.1.11
default via 10.0.1.1 dev eth1 table 1
10.0.1.0/24 dev eth1 src 10.0.1.15 table 1

# file route-eth2
# Assume your 2nd ENI IP address is  10.0.1.12
default via 10.0.1.1 dev eth1 table 2
10.0.1.0/24 dev eth2 src 10.0.1.12 table 2

Then you follow the document you mentioned instruction, in /etc/sysconfig/network-scripts, create a rule-ethX , i.e. rule-eth0, rule-eth1 1. Increment the table number to match route-ethX 2. Change the IP to the assigned internal network address of the ENI.

# file rule-eth0 ,but as the doc suggest, you should skip this file.
# format : from  ENI_IP/CIDR table <table_number> 
from 10.0.1.10/32 table 0

#file rule-eth1
from 10.0.1.11/32 table 1

You should play the Centos network setup with your Local Vmware/virtualbox virtual network adapter. Then you don't need to worry a sudden

Answer 2

The answer by @mootmoot got me most of the way on this one but, just for completeness, here are the steps that eventually got me all the way to a working configuration. I'm sure there are better ways to do this but in case someone else is stuck ...

I made the following modifications to the files in /etc/sysconfig/network-scripts

I left the interface defined in ifcfg-eth0 alone and added aliases as follows:

ifcfg-eth0:0

DEVICE=eth0:0
BOOTPROTO="static"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
PEERDNS="yes"
IPV6INIT="no"
DEFROUTE="no"


IPADDR0="10.0.1.XXX"
PREFIX0="24"

Where 10.0.1.XXX is the first secondary address on the ENI (the primary address remains configured by dhcp as specified in the default ifcfg-eth0 file)

continue adding ifcfg-eth0:X files until all the ips in the ENI are specified.

Then define the second device in the file ifcfg-eth1 using the same template as for the eth0 aliases above and the primary IP address of the ENI.

Then add aliases for eth1 by defining files called ifcfg-eth1:0 etc as above.

define the following routes files:

route-eth0

default via 10.0.1.1 dev eth0  table 1

route-eth1

default via 10.0.1.1 dev eth1  table 2

where 10.0.1.1 is the address of the VPC gateway

I then needed to define rules for all IPs in eth2 and all secondary ips in eth1 as follows:

rule-eth0

from 10.0.1.106 lookup 1
from 10.0.1.105 lookup 1
... etc

rule-eth1

from 10.0.1.41 lookup 1
from 10.0.1.226 lookup 1
... etc   

I also added the line

GATEWAYDEV=eth0

to /etc/sysconfig/networking

Hope that helps someone in a similar situation, thanks for all the advice.

Answer 3

First, I will ALWAYS check AWS EC2 Network interface limit. (ref : http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI)

For example , if you create c1.medium instance, you cannot use more than 2 ENI and exceed 6 IP addresses per interface. If you try to create 20 virtual network interface in the OS, it will fail.

I am not sure the reason that you want to bound 1 Public IP to 1 internal IP, unless you intend to move to IPv6, put ELB in front your EC2 instance to wrap the IPv6 to IPv4. (Because EC2 doesn't support IPv6 )

But looking at the way of AWS VPC setup, you cannot think like using Centos in a Physical server box, which you can make something flexible like eth0:0, eth0:1, ...,eth0:20,

Unfortunately, this is not the case of AWS ENI and network interface. It seems AWS assign unique MAC for individual "private ip addresses". So limited private IP is tied to an interface. As stated above, you can only c1.medium create 2 network interface, each with 6 private IP max.
(ref : http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/MultipleIP.html)

You need to think as in AWS VPC ways, not the flexible way of physical machine may give. Like some extreme case :

• VPC : The CIDR and instance IP address must be related. E.g. 10.1.0.0/19. The connection relies on the AWS VPC route tables.
• Physical Server and network : You can use multiple CIDR to all the machine. E.g. a Server can have 1 network interface with IP adderss from two and more CIDR , e.g. 10.1.0.0/19, 172.16.0.0/19 . eth0:1 10.1.0.15 eht0:2 172.16.7.15. You can setup/add/delete your own route.

It seems you need to redesign your network infra according to AWS VPC ways.

You may ask, what if you think about Elastic IP? Well, EIP limit for VPC are 5. And do remember this, Public IP address are scarce. Perhaps splitting the application usage with multiple-domains is a better idea.

(Update) Associate Public IP address to Private IP. (Try to use AWS cli or aws sdk script so you can recreate the instance with a script)

1. Create EC2 instance , assign multiples private IP to each Network interface
2. Go to VPC - Elastic IP , allocate Elastic IP
3. Go to the newly allocated elastic IP, select associate, there is option to attach to specific EIP and Private IP address.

Since aws set EIP limit to 5 for each VPC, you need to check whether the soft limit has been raise. To test it out, just try allocating 18 EIP, and point to individual Private IP address.

You cannot assign EIP inside Centos instance. EIP options is found under EC2 and/or VPC console.