0

I want to write images to aws s3. As a video plays i am trying to process images through some functions and then when its done I wish to store it to a specific path. imageio directly checks the extension in the name and writes the image for the appropriate file format.

s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket)
obj = bucket.Object(filepath+'/'+second+'.jpg')
img.imwrite(obj)

If I were to write this to a local location and then write it to s3 then it works but is there a better way where I could store it to s3 without having to write it locally.

Any help is appreciated.

Akshay Hazari
  • 3,186
  • 4
  • 48
  • 84

1 Answers1

3

You can use something like BytesIO from python's io package, to create the file object in memory, and pass that to the boto3 client, like this:

from io import BytesIO

s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket)
in_memory_file = BytesIO()
img.imwrite(in_memory_file)
obj = bucket.Object(filepath+'/'+second+'.jpg')
obj.upload_fileobj(in_memory_file)

This should solve the problem, without having to write the file to disk.

Iberê
  • 1,211
  • 1
  • 11
  • 16
  • For anyone intersted, the inverse operation to read a file from S3 into image in memory is to use `s3.download_fileobj()` write the result into a `BytesIO` file object, and then do `obj.read()` to get the bytes. These bytes can then be passed directly into `imageio.imread(bytes)` – Adam Hughes May 07 '20 at 18:55