Azure Function Memory Exceeded

Question

I wrote a Azure function in python to unzip a file and upload it back to blob storage. It works great for small files but fails when I try gb files.

I know the issue is that the file is too large to put into memory and needs to be streamed. Any suggestion on how to stream the unzip and upload of this file

My code:

import azure.functions as func
import json
import logging
import os
from zipfile import ZipFile
from azure.storage.blob import BlobServiceClient, ContainerClient, blob

def main(mytimer: func.TimerRequest) -> None:

    source_conn_str = xxx
    source_container = xxx
    blob_service_client_origin = BlobServiceClient.from_connection_string(source_conn_str)
    source_fileName = xxx
    blob_to_copy = blob_service_client_origin.get_blob_client(container=source_container, blob=source_fileName)
    
    # Step 2. Download zip file to local tmp directory 
    os.chdir('/tmp/')
    print("Downloading file")
    blob_data = blob_to_copy.download_blob()
    data = blob_data.readall()
    print("Download complete")
    
    # Step 3. Save zip file to temp directory
    local_filepath = xxx
    with open(local_filepath, "wb") as file:
        file.write(data)
    
    # Step 3. Unzip file to local tmp directory 
    with ZipFile(local_filepath, 'r') as zipObj:
        zipObj.extractall()
    
    
    # Step 4. Upload file to storage account 
    dest_conn_str = xxx
    blob_service_client = BlobServiceClient.from_connection_string(dest_conn_str)
    container_name = xxx
    
    #Set the local file name 
    local_file_name = xxx
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=local_file_name)

    # Upload the file to blob storage
    print('Uploading file')
    with open(local_file_name, "rb") as data:
        blob_client.upload_blob(data, overwrite = True)
    print('File Upload complete')`enter code here`

When I run the Azure Function it returns: Exception message: python exited with code 137. Which means it's out of memory. Any suggestions are extremely appreciated.

score 1 · Answer 1 · answered Feb 28 '22 at 01:18

Please check if given references and Work arounds help to:

When I run the Azure Function it returns: Exception message: python exited with code 137. Which means it's out of memory.

Yes, in the Python Azure Function App, Error Code as 137 means Out-of-Memory.

Workaround 1:

Please visit the MS Doc Service limits in Hosting Plans of Function App.

As it given 137 Error Code and to check if it is timeout issue, keep an eye on Azure Application Insights logs when the function app is executed.

If you find like "Timeout value of xx:xx:xx exceeded by function", then research about below points:

Check the code execution is taking more time. If yes, improve the code or/and increase the default timeout value.

To change the function timeout value from the host.json, refer this MS Doc for more information.

functionTimeout indicates the timeout duration not for the single function but for all the functions.
Update the default timeout value and test the function again.
Also, Please refer the Memory profiling on Python Function apps to understand more about managing the memory in our functions.

Workaround 2:

Please refer this solution as it is said that, Python error code 137 means the process was killed by OS, most like due to excessive memory usage.

You can check the memory usage of your app from the portal using metrics as referenced practically in SO Thread1 and SO Thread2.

Workaround 3:

It can be like maximum number of connections exceeded than the given value to the blob storage account. For the larger file uploads via Azure-python-functions, max_connections value will be provided in greater than 1 which enables the ability to download the file in chunks and writing them to the stream.

Please check if this reference helps to.

Please check the below references containing the code of unzipping the files from azure blob storage with downloading, without downloading as well as unzipping the password protected zip file automatically:

Azure Function Memory Exceeded

1 Answers1