0

I would like to use AWS Lambda (Python) for Image (svs) pre-processing (create tiles ect). Unfortunately the images are around 1 GB and wont fit on /tmp (512MB). Hence, I was hoping to load the images either directly into RAM through something like:

s3_response_object = s3_client.get_object(Bucket=bucket, Key=key)
object_content = s3_response_object['Body'].read()
inmemoryfile = io.BytesIO(object_content)
Image.open(inmemoryfile)

or download a picture directly to ramfs or something similar: 

from memory_tempfile import MemoryTempfile 
import memory_tempfile

also only 1 out of 10 level is needed in the svs file. Hence, if there is a way to only read specific information from the file from a s3 bucket - that would be great.

thanks

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
y4nnick
  • 101
  • 1
  • 4
  • Did you try either of these options? What happened? On the question of reading specific parts of an object in S3, yes you can do this with ranged gets as long as you know the offset and length of the part you want (you may need to retrieve this info first from some header record within the file itself). – jarmod Dec 05 '19 at 16:03

1 Answers1

0

I recommend s3fs https://github.com/dask/s3fs

import s3fs
fs = s3fs.S3FileSystem()

# You don't need "s3://" in your path if you don't want it
with fs.open('s3://file/path') as fh:
    do_stuff(fh)
Michael Silverstein
  • 1,653
  • 15
  • 17