How to access AWS S3 with KMS from org.apache.hadoop.fs.s3a.S3AFileSystem API

Question

I have EC2 configured with

  <property>
     <name>fs.s3a.aws.credentials.provider</name>
    <value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value>
  </property>
 <property>
    <name>fs.s3a.server-side-encryption-algorithm</name>
    <value>SSE-KMS</value>
  </property>
 <property>
    <name>fs.s3a.server-side-encryption.key</name>
    <value>arn:aws:kms:zz-jjbbcc-1:123432:key/AABBCC</value>
  </property>

with this configuration in core-site.xml I am easily able to put files and create folders (via aws cli) in S3 without giving any authentication details as I have already configured in core-site.xml

Now I want to access the S3 bucket and create folder by using S3AFileSystem, But when i am calling

S3AFileSystem fs  
fs.mkdirs(somepath); // this will create folder in S3. 
Its throwing Accessdenied 403 Exception. 


java.nio.file.AccessDeniedException: s3a://xxx-xxx/xxx/.FolderIwantToCreate: innerMkdirs on s3a://xxx-xxx/xxx/.FolderIwantToCreate: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XXXXXXX; S3 Extended Request ID: xxxxxxxxxxxxx=), S3 Extended Request ID: /XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:174) ~[hadoop-aws-2.9.2.jar:?]
    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:117) ~[hadoop-aws-2.9.2.jar:?]
    at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1683) ~[hadoop-aws-2.9.2.jar:?]
    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216) ~[hadoop-common-2.9.2.jar:?]

But while debugging, I tried

((S3AFileSystem) fs).delete(path,true)

where path has s3://MyBUCKET/SOMEFOLDER/ surprisingly SOMEFOLDER gets deleted

Did i miss any configuration? S3(enabled with KMS) and doesn't have any bucket policy, IAM has Full Access on S3.

EC2 has IAM Role which has the following policies:
S3 Access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

KMS Policy :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:CreateAlias",
                "kms:CreateKey",
                "kms:DeleteAlias",
                "kms:Describe*",
                "kms:GenerateRandom",
                "kms:Get*",
                "kms:List*",
                "kms:TagResource",
                "kms:UntagResource",
                "iam:ListGroups",
                "iam:ListRoles",
                "iam:ListUsers"
            ],
            "Resource": "*"
        }
    ]
}

S3 Bucket Don't have any Policies.

Do you have a role attached to your EC2 instance? Does it have permissions to access S3? Not sure about S3AFileSystem per se, but you should make sure that it is able to automatically collect credentials (e.g., like s3 client is) or to pass the credentials manually... — StefanN, Jun 17 '21 at 07:58
Can you also post your relevant IAM and S3 policies? Also how do you pass the credentials to your S3 client? — StefanN, Jun 18 '21 at 07:25
@StefanN Updated the question and I'm not passing the credentials to your S3 client, I am expecting it to take automatically (as was cli) — LMK IND, Jun 18 '21 at 14:19
Please format the code snippets properly! Very hard to read, but I saw an error in your IAM policy. The correct way to specify action is `"Action": "s3:*"` (your policy is missing the "*") — StefanN, Jun 18 '21 at 14:59
@StefanN Sorry for not formatting, i just formatted and updated — LMK IND, Jun 18 '21 at 17:36

score 1 · Answer 1 · answered Jun 27 '21 at 21:28

I would look at your KMS Key policy. There are two things I see:

Key policies are 'king'. They override even IAM policies when it comes to that particular key. This key policy lacks the ability to use IAM to delegate/give KMS permissions on this key. With this policy, even if an IAM policy would give permission to use this particular KMS key, they would be ignored although there would be no indication of that from the IAM service. See the example under 'Default Key Policy' here for more information: https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html
If you are storing keys in S3, I think you need kms:GenerateDataKey, and possibly kms:decrypt permission depending on how the service will verify it successfully wrote the file. Sometimes, upon setting this up the underlying service will quickly write and read back a temporary file you never see to make sure the permissions are in place. See somewhat related link here for what S3 permissions are needed from KMS when using that service for CMKs: https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-access-default-encryption/

How to access AWS S3 with KMS from org.apache.hadoop.fs.s3a.S3AFileSystem API

1 Answers1