Limit programmatic GET requests to AWS S3 buckets to prevent accidental bucket increases / costs

Question

I have a bucket containing many files with JSON. For compliance reasons, the bucket is open to the public. We want to try to put something in place to limit the requests to the bucket so that we don't get charged extra if someone's code goes wrong and puts through a lot of requests. I thought I could use API gateway, but that would assume that we know who is requesting and would be able to throttle their access, but since this is open to the public that wouldn't work. Any other recommendations for limiting access to a bucket to prevent abuse either intentional or unintentional?

Answer 1

One approach to limit access to your public bucket containing JSON files is to use Amazon CloudFront, which is a content delivery network that can cache the files and provide a layer of security between the user and the bucket. CloudFront acts as a proxy between the user and the S3 bucket, and can help reduce the number of requests going directly to the bucket, which can help prevent abuse and reduce costs.

To use CloudFront with your S3 bucket, you would create a CloudFront distribution that pulls the files from your S3 bucket and caches them at edge locations around the world. This can help reduce latency for users, and reduce the load on your bucket. You can then configure the distribution to restrict access to the files, such as by using signed URLs or signed cookies.

Signed URLs are unique URLs that grant time-limited access to a specific file in your S3 bucket. You can generate these URLs using AWS IAM credentials or using CloudFront's key pairs. When a user requests a file, the CloudFront distribution checks the validity of the signed URL before serving the file. This approach can help prevent unauthorized access to your files, as only users with a valid signed URL can access them.

Signed cookies work in a similar way to signed URLs, but they are stored as browser cookies rather than as part of the URL. This approach can help reduce the risk of URL tampering, as the signed information is stored in a secure cookie.

Another approach to limit access is to use AWS WAF (Web Application Firewall), which can help protect your bucket against common web attacks, such as SQL injection or cross-site scripting (XSS) attacks. You can create rules that block requests that match certain patterns, such as requests with a high number of requests or requests with a specific user agent. This can help prevent abuse by malicious actors.

Amazon CloudFront and AWS WAF can help reduce the load on your public bucket containing JSON files, and limit access to authorized users. Signed URLs and signed cookies can help prevent unauthorized access, while WAF can help protect against common web attacks.