AWS - NAT between several VPC

Question

In AWS, I have several VPC. In each VPC, I have an EC2 instance, running a server. For the moment, each EC2 has an Elastic IP because my servers are dedicated for IoT, and our connected objects need a dedicated IP address, they cannot use DNS. And of course each VPC is for a different customer.

Here is a very simple diagram of the architecture:

Of course I have also all the dedicated network to reach the server, that means the security groups, internet gateway etc. But always one for a dedicated VPC.

But on AWS, there is a limitation of 5 Elastic IP per region. Because of that, I need to find a solution to not use am Elastic IP for each instance.

What solution can I use on AWS for that ? The simple way would be to create a NAT, using one single EIP and redirecting to the correct server using port. Something like that:

But the problem is that I'm using different VPC.

How can I do that between different VPC ?

(I'm mentionning NAT because it's the solution I know outside "aws world", but there may be other solutions, like NAT Gateway, NAT Instance, Transit Gateway, Internet Gateway etc.. I'm a bit lost)

Thanks

Answer 1

You can use VPC peering between the VPC with a NLB in a frontend VPC and the other VPCs.

A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs, or with a VPC in another AWS account. The VPCs can be in different regions (also known as an inter-region VPC peering connection).

Answer 2

One VPC per customer is not going to scale for long if you're successful. This is a soft limit (i.e. it can be raised) of 5 VPCs per region per account, I don't know how high AWS will raise it.

There's a soft limit of 5 elastic IPs per account. I assume AWS would be willing to raise this, but IPv4 is a scare resource so I doubt they would give you a huge number of them. Of course each instance gets a public IP address, but it changes if you stop / start the instance.

NAT is for egress, not ingress. I don't think it's going to help you. If you can use a single IP with each customer on a specified port, proxied through a central account, that would be efficient and would scale. You may have problems with firewalls though, and it means you may pay for traffic twice - once into the proxy, then once out to the VPC / account that services it.

Transit Gateway

Transit gateway provides connectivity to virtually any number of AWS VPCs, in one or many accounts. It's easier than VPC peering as it's a hub and spoke architecture rather than point to point.

If you don't want your accounts taking to each other, just to a central account, you can put firewalls in place for account isolation. I use NACLs for that, reserving security groups for tiering (web, application, db, etc). I've largely automated NACL creation using CloudFormation for my current project - we deploy virtually everything as code.

Transit Gateway charges per VPC attachment. From memory it's a bit more expensive than VPC peering, but VPC peering might charge for data both being sent and received which would make it similar. I haven't looked into this in detail because enterprises don't really care about the cost of bandwidth, within reason.

Your requirement to use one Elastic IP to be delivered to multiple VPCs cannot be done by Transit Gateway. That would probably have to be done by something like Nginx running in an instance, though maybe you can look at ALB / NLB for this. I don't have time to fully consider your requirements and create a solution at the moment.

Ok Solution

IPv6 has a very large number of addresses. If your clients can support IPv6 that's a solution that will scale. You don't need elastic IPs for IPv6, you allocate an IPv6 range to your VPC and its yours with no NAT.

You're still going to run into the VPCs per account limit with this option.

Better Solution

A multi-account architecture with one account per customer would be easier to scale and would avoid the limits. Use AWS Control Tower to manage the accounts. Transit Gateway can do inter-account networking for you, but beware of costs.

Multi-account is not particularly difficult to manage. You set up an AWS organisation and your billing information goes in there - credit card or you can arrange monthly invoices with AWS. You can create new accounts either with AWS Organisations (ok) or if you use Control Tower it has an account factory. The advantage of Control Tower is it puts all kinds of best practice security in place for you. This disadvantage is it takes an hour to create a new AWS account as it has to roll out all the controls to each new account.

Best Solution

I would suggest the best solution is to write your application to service multiple customers through one account / VPC / IP if your clients can do that. Use some kind of authentication / authorisation service to ensure requests are associated with the correct customer.