Unable to write or read from S3 bucket with Default AWS KMS encryption enabled

Question

I am unable to read or write into a Default AWS KMS encrypted bucket without using the following configuration on my Qubole cluster

fs.s3a.server-side-encryption-algorithm=SSE-KMS
fs.s3a.server-side-encryption.key=<key>

But if I enable this configuration, it's only writing encrypted data into all the other buckets (which do not have encryption enabled) as well.

Is there a work around for this?

Answer 1

You can try the followings:

1. Remove the following from cluster overrides:

fs.s3a.server-side-encryption-algorithm=SSE-KMS 
fs.s3a.server-side-encryption.key=<key>

Reference: https://docs.qubole.com/en/latest/admin-guide/enabling-encryption/enabling-sse/enabling-sse-kms.html

2. Use default encryption as mentioned here:

https://docs.aws.amazon.com/AmazonS3/latest/user-guide/default-bucket-encryption.html

3. Restart the cluster

Ideally after following the above steps, you should be able to achieve your goal.