Azure Function (Python) w/ Storage Upload Trigger Fails with Large File Uploads

Question

Azure Function (Python) triggered from file uploads to Azure Storage. Function works fine for files up to ~120MB. I just load tested with a 2GB file and the function produced the error Stream was too long.

Where is this limitation documented?
How would I overcome it using Python?

Using boto3 library to PUT files to AWS S3

def main(myblob: func.InputStream):
        logging.info(f"Python blob trigger function processed blob \n"
                                 f"Name: {myblob.name}\n"
                                 f"Blob Size: {myblob.length} bytes")

        myblobBytes = myblob.read()

        fileName = pathlib.Path(myblob.name).name

        s3 = boto3.resource(
                's3',
                aws_access_key_id="youguessedit",
                aws_secret_access_key="noyoudidnt",
        )

        response=s3.Bucket(bucketName).put_object(Key="folder/" + fileName, 
        Body=myblobBytes, ContentMD5=md5Checksum)
        
        response.wait_until_exists()

What library/sdk are you using in your Azure Function? According to [this answer](https://stackoverflow.com/a/41233653/7867968) certain functions are not efficient for large file transfers into Azure Blob — C.Nivs, Sep 11 '20 at 14:46
Azure Python SDK. I saw that post. Looks like it focuses on .NET. — ericOnline, Sep 11 '20 at 15:19
yes, but what does your code look like? The concept is probably the same — C.Nivs, Sep 11 '20 at 15:21

score 0 · Accepted Answer · answered Oct 09 '20 at 03:12

0

Changed boto3 from put_object to upload_fileobj and setup TransferConfig for multipart_threshold=1024*25, max_concurrency=10, multipart_chunksize=1024*25, use_threads=True.

Rips now!

Able to transfer 2GB in 89secs! Not bad.

answered Oct 09 '20 at 03:12

ericOnline

1,586
1
19
54

Azure Function (Python) w/ Storage Upload Trigger Fails with Large File Uploads

1 Answers1

Linked