How to use Python smart_open module to write to S3 with server-side encryption

Question

In older versions (<5.0.0) of the Python smart_open module you could write to an S3 location with server-side encryption like this:

import smart_open
writer = smart_open.open("s3://bucket-name/path/to/file.txt", 'w', transport_params={'multipart_upload_kwargs':{'ServerSideEncryption': 'AES256'}})
writer.write("nothing to see here\n")
writer.close()

However, the underlying mechanisms changed with version 5.0.0 (see here). The old way to doing it no longer works. It acts like I did not pass the encryption parameter:

ValueError: the bucket 'bucket-name' does not exist, or is forbidden for access (ClientError('An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: Access Denied'))

I find the documentation to be confusing and insufficient to figure it out myself. How can I write to S3 with encryption using newer versions of smart_open?

Answer 1

Got a very quick and helpful response from the smart_open team. Here is how it works in smart_open version 5.0.0

import smart_open
client_kwargs = {'S3.Client.create_multipart_upload': {'ServerSideEncryption': 'AES256'}}
writer = smart_open.open("s3://bucket-name/path/to/file.txt", 'w', transport_params={'client_kwargs': client_kwargs})
writer.write("nothing to see here\n")
writer.close()

With that simple change it works as great as ever!