1

I have a Python Azure Functions timer trigger that is run once a day and archives files from a general purpose v2 hot storage container to a general purpose v2 cold storage container. I'm using the Linux Consumption plan. The code looks like this:

container = ContainerClient.from_connection_string(conn_str=hot_conn_str, 
                                                   container_name=hot_container_name)
blob_list = container.list_blobs(name_starts_with = hot_data_dir)
files = []
for blob in blob_list:
    files.append(blob.name) 
for file in files:
    blob_from = BlobClient.from_connection_string(conn_str=hot_conn_str, 
                                             container_name=hot_container_name, 
                                             blob_name=file)
    data = blob_from.download_blob()
    blob_to = BlobClient.from_connection_string(conn_str=cold_conn_str, 
                                             container_name=cold_container_name, 
                                             blob_name=f'archive/{file}')
    try:                                         
        blob_to.upload_blob(data.readall())
    except ResourceExistsError:
        logging.debug(f'file already exists: {file}')
    except ResourceNotFoundError:
        logging.debug(f'file does not exist: {file}')
    container.delete_blob(blob=file)

This has been working for me for the past few months with no problems, but for the past two days I am seeing this error halfway through the archive process:
The operation has timed out.
There is no other meaningful error message other than that. If I manually call the function through the UI, it will successfully archive the rest of the files. The size of the blobs ranges from a few KB to about 5 MB and the timeout error seems to be happening on files that are 2-3MB. There is only one invocation running at a time so I don't think I'm exceeding the 1.5GB memory limit on the consumption plan (I've seen python exited with code 137 from memory issues in the past). Why am I getting this error all of a sudden when it has been working flawlessly for months?
Update
I think I'm going to try using the method found here for archival instead so I don't have to store the blob contents in memory in Python: https://www.europeclouds.com/blog/moving-files-between-storage-accounts-with-azure-functions-and-event-grid

ddx
  • 469
  • 2
  • 9
  • 27
  • How about set the function timeout? – Cindy Pau Nov 19 '20 at 01:28
  • Have you made any changes on your function configuration these days ? – Hury Shen Nov 19 '20 at 02:36
  • The function timeout is set to 10 minutes already and the operation timeout happened after about 5 minutes. @Hury Shen, no I haven't made any configuration changes. – ddx Nov 19 '20 at 05:49
  • May I know if you added new functions(such as http trigger function or new timer trigger function) in the same function app recently ? – Hury Shen Nov 19 '20 at 05:56
  • @Hury Shen, no new functions have been added – ddx Nov 19 '20 at 06:15
  • Can you see the error was caused by which line of your code ? – Hury Shen Nov 19 '20 at 06:17
  • It looks like the error occurred on either one of the `BlobClient.from_connection_string()` calls or on `data = blob_from.download_blob()`. I did change the code to use `start_copy_from_url()` like the article I posted mentioned and it did work this morning and seems to be much faster. – ddx Nov 19 '20 at 16:14
  • It might have also failed on `blob_to.upload_blob(data.readall())` if the timeout happened before an exception was caught, but I'm suspecting it happened at `data = blob_from.download_blob()` – ddx Nov 19 '20 at 17:31
  • So does `start_copy_from_url()` solve all of your problems ? – Hury Shen Nov 20 '20 at 06:16
  • It looks like `start_copy_from_url()` has been working, thanks! I also made a post on the Azure Q&A forums and an employee from Microsoft reached out to look into the `The operation has timed out.` issue – ddx Nov 20 '20 at 20:35
  • So could you please provide an answer under this post which summarize the workaround of `start_copy_from_url()` for other communities reference ? – Hury Shen Nov 23 '20 at 04:19
  • Or do you mind if I summarize the workaround of `start_copy_from_url()` below as answer for other communities reference ? – Hury Shen Nov 25 '20 at 01:30

1 Answers1

1

Just summarize the solution from comments for other communities reference:

As mentioned in comments, OP uses start_copy_from_url() method instead to implement the same requirements as a workaround.

start_copy_from_url() can process the file from original blob to target blob directly, it works much faster than using data = blob_from.download_blob() to store the file temporarily and then upload data to target blob.

Hury Shen
  • 14,948
  • 1
  • 9
  • 18