1

I have some EC2 instances in a private subnet (as part of a larger EKS cluster) in Account A. There is a containerized application running on these EC2 instances that reaches out to S3 for some data to process. The S3 bucket is in Account B. Currently the application just uses plain Access Key and Secret Key (since its a legacy app we don't want to modify).

Currently I have a NAT gateway configured that allows the EC2 instances to access the internet. When the applications (within the Kubernetes pods) reaches out for files in S3, it goes through the NAT gateway, over the internet.

This incurs a significant cost for data transfer processed via the NAT gateway.

I've heard that VPC Gateway endpoints for S3 can help alleviate the problem by allowing access to go through the Amazon network, without using the internet. This can save money when it comes to data transfer costs. However, most blog posts seem to assume that the EC2 instances and the S3 buckets exist in the same account and region. In my case, they are in different AWS accounts.

I've also heard that VPC Interface Endpoints are an option. The documentation mentions they can be used across regions (but doesn't mention across accounts). More importantly, there is a cost associated with the data transfer, unlike with VPC Gateway endpoints.

Is this the appropriate tool to use for this case or is there an alternate way that I can access an S3 bucket from a separate AWS account while avoiding too many data transfer costs?

My question is similar to this one, but that question focuses on public IP addresses, and in my case all my EC2 instances are in a private subnet.

The Unknown Dev
  • 3,039
  • 4
  • 27
  • 39
  • Possibly https://stackoverflow.com/questions/64371121/does-aws-has-cross-account-data-transfer-cost-if-both-service-are-in-same-region? I've done similar cross-account bucket copies in the same region without incurring costs beyond the typical operation costs (PUTs), for ~1 PB of data. But before you do anything, I'd open a support ticket with AWS to confirm, or, if you have an account manager, reach out to them. – wkl Aug 16 '22 at 20:12
  • **Side-note:** The term 'Data Transfer' is typically associated with data going from AWS to the Internet, which does not apply when resources are communicating in the same Region. In the case of **NAT Gateway**, the charges are "per hour" and "per GB data processed". It would appear that you are wanting to reduce these "per GB data processed" charges for NAT Gateway. – John Rotenstein Aug 17 '22 at 00:01

2 Answers2

3

An S3 vpc gateway endpoint is definitely what you want to be using. It doesn't matter if the S3 bucket you want to access is in another account, it will work. The gateway endpoint gives connectivity to the S3 service, so it doesn't matter where the bucket you want to access lives.

Do note that the gateway endpoint should be created in the same region as that where the S3 bucket lives.

Paolo
  • 21,270
  • 6
  • 38
  • 69
  • Thanks! I went ahead and created a VPC Gateway endpoint and associated to my private subnet route tables. It still seems like theres data being processed via the NAT gateway. Is there any sort of priority in the routing rules I should be aware of? (I would like to keep the NAT route entry for other non-S3 internet connectivity). – The Unknown Dev Aug 17 '22 at 21:13
  • @TheUnknownDev *It still seems like theres data being processed via the NAT gateway.* How did you check this? – Paolo Aug 17 '22 at 21:23
  • I just reviewed CloudWatch metrics for my NAT gateway. There are metrics like `BytesOutToDestination` that I observed increased while I was running my workload, the same as they did when I ran the workload without the VPC Endpoint Gateway. – The Unknown Dev Aug 17 '22 at 23:31
  • @TheUnknownDev what does your workload do? Does it only talk to s3? – Paolo Aug 18 '22 at 06:03
  • Yes. Essentially it pulls down large data files from S3, transforms them, and uploads a new file with transformed data. – The Unknown Dev Aug 18 '22 at 14:16
  • @TheUnknownDev are there other instances that use the NAT gateway? are you sure you configured the routing properly? try running vpc reachability analyzer from the instance to the vpc endpoint – Paolo Aug 18 '22 at 14:34
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/247367/discussion-between-the-unknown-dev-and-paolo). – The Unknown Dev Aug 18 '22 at 15:00
2

I am assuming that you are wanting to reduce the "per GB data processed" costs of NAT Gateway. (Note, this is different to "Data Transfer", which is a term normally associated with sending traffic from AWS to the Internet).

Please note that there is no charge for sending traffic "between AWS Accounts". AWS Accounts are merely constructs for billing and security and do not impact architecture. Rather, charges arise from transferring data in/out of VPCs and between Regions.

In your scenario, resources in Amazon EKS in a VPC are communicating with S3 through NAT Gateway. Therefore, options are:

Option 1: Use a NAT Instance

A NAT Instance is just an Amazon EC2 instance that is acting as a NAT. It is charged purely per-hour. However, the size of the EC2 instance could restrict the network bandwidth available.

Option 2: Use a VPC Endpoint for S3

You can create a Gateway endpoint for Amazon S3 - Amazon Virtual Private Cloud, which creates a direct link between the VPC and the Amazon S3 service in the same region. This means requests to S3 do not go via the NAT Gateway.

The documentation says:

You can access Amazon S3 from your VPC using gateway VPC endpoints. After you create the gateway endpoint, you can add it as a target in your route table for traffic destined from your VPC to Amazon S3.

There is no additional charge for using gateway endpoints.

This would be the better option for the scenario you have described.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Thank you for clarifying the difference between "data transfer" and "data processed". You are correct, that I'm concerned with "data processed" in this case. – The Unknown Dev Aug 17 '22 at 01:08