Upload file from memory to S3

Question

I have downloaded a csv file from S3 into memory and edited the file using Boto3 and Python. How can I can reupload this file to S3 without ever storing it locally?

What have you tried so far? Are there any functions in the Boto3 docs that looked promising but did not fit your use case for some reason? — James Mchugh, Nov 21 '19 at 00:59
I don't know how Boto3 sends files but Python has `io.StringIO` and `io.BytesIO` to create file-like object in memory so it can be used as opened file - it has functions like `read()`, `seek()`, `write()`, `flush()` etc. — furas, Nov 21 '19 at 01:10
From a quick look, I can see the [put_object](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.put_object) method from the `Client` class which should do exactly as you asked. — James Mchugh, Nov 21 '19 at 01:27
The `put_object` method worked, thank you James. Sorry for the terrible question — Ciaran, Nov 21 '19 at 02:54
@Ciaran No problem. I know that searching through docs when learning a new technology, API, etc. can be daunting. However, it is quite rewarding to get familiar with those docs ASAP. In the future, if you have other questions like this, I would strongly recommend searching through the docs a bit first. If you still have the question afterwards, feel free to ask the SO community, just make sure you mentioned what you have looked into and why what you found hasn't worked for you. — James Mchugh, Nov 21 '19 at 14:55

score 9 · Answer 1 · answered Nov 21 '19 at 02:42

9

As per @JamesMchugh, from put_object():

response = client.put_object(
    Body=b'bytes'|file,
    Bucket='string',
    Key='string',
)

answered Nov 21 '19 at 02:42

John Rotenstein

241,921
22
380
470

Is there a faster way than this? I am trying to upload a file of size approx 2GB. Other functions such as upload_file needs to store the file locally first. – user3807691 Sep 07 '21 at 10:42

score 5 · Answer 2 · edited Mar 05 '21 at 11:47

In my case, I have a list of dictionaries and I have to create in memory file and save that on S3. Following Code works for me!

import csv
import boto3
from io import StringIO
# input list
list_of_dicts = [{'name': 'name 1', 'age': 25}, {'name': 'name 2', 'age': 26}, {'name': 'name 3', 'age': 27}]

# convert list of dicts to list of lists
file_data = []
header = list(list_of_dicts[0].keys())
file_data = [[d[key] for key in header] for d in list_of_dicts]
file_data = [header] + file_data

# create in memory file and write data to it.
file_to_save = StringIO()
csv.writer(file_to_save).writerows(file_data)
file_to_save = bytes(file_to_save.getvalue(), encoding='utf-8')
file_name_on_s3 = 'my_data.csv'

# save in memory file to S3
client = boto3.client('s3',
                      aws_access_key_id='your access key',
                      aws_secret_access_key='your secret key')
response = client.put_object(
    Body=file_to_save,
    Bucket='your bucket name',
    Key=file_name_on_s3,
)

Is it necessary to encode the file before saving? Can it not be saved as it's written by csv.writer? — thleo, Nov 18 '21 at 22:12

Upload file from memory to S3

2 Answers2