How to access an S3 object from a remotely executed fargate task?

Question

I've written a java program to read from an S3 bucket, that's running in a Fargate task on AWS, but is failing with an AccessDeniedException The program is in a jar file that is being executed remotely using ECS Exec The task role for the Fargate task has the following permissions, but the following lines still fail with this below exception

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectacl",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::*"
        }
    ]
}

ListObjectsV2Request req = new ListObjectsV2Request().withBucketName("myBucket").withMaxKeys(200);
ListObjectsV2Result result = amazonS3.listObjectsV2(req);

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 0Z58XY0CF91Q6H4Y; S3 Extended Request ID: sVoHHMEM4+5ti+MlcaHsgvJxyvbFlJrMVZMC4cBOFI1hPg/QtsKExqFiqNECH2ZKoUXuJMYwAt0=), S3 Extended Request ID: sVoHHMEM4+5ti+MlcaHsgvJxyvbFlJrMVZMC4cBOFI1hPg/QtsKExqFiqNECH2ZKoUXuJMYwAt0=
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4914)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4860)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4854)
    at com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:923)
    at com.mycompany.ReadS3Object.getFilesOnS3(ReadS3Object.java:263)
    at com.mycompany.ReadS3Object.run(ReadS3Object.java:141)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
    at picocli.CommandLine.access$1300(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
    at picocli.CommandLine.execute(CommandLine.java:2078)
    at com.mycompany.ReadS3Object.main(ReadS3Object.java:132)

What needs to be changed to allow the listObjectsV2 call to complete?

EDIT

So it turns out the issue was that the bucket the code was trying to list doesn't exist! When I ran the code locally, it gave the correct bucket not found exception, but running it remotely gave the AccessDeniedException. Creating the bucket resolved the problem

score 1 · Answer 1 · answered Apr 07 '21 at 16:42

Looks like your S3 V1 code is missing creds that references an IAM user that has permission to invoke S3 operations. As a test, provide that role full S3 access and see if this issue still occurs.

Also as a general recommendation, you should consider moving away from the Older V1 Amazon S3 API and consider using V2 - which is the version that Amazon recommends using when writing Java code to invoke Amazon S3 operations.

When you see the use of the new operator like this:

ListObjectsV2Request req = new ListObjectsV2Request().withBucketName("myBucket").withMaxKeys(200);

it's V1 code. V2 code does not use the new operator to create objects, but instead uses a builder():

ListObjectsRequest listObjects = ListObjectsRequest.builder()
                    .bucket(bucketName)
                    .build();

thanks, I've added the permissions for the role to the question — Joseph McCarthy, Apr 07 '21 at 17:13
also, thanks for the advice on the V2 code, but for now I'm restricted to the V1 libraries — Joseph McCarthy, Apr 08 '21 at 14:57

How to access an S3 object from a remotely executed fargate task?

EDIT

1 Answers1