4

I'm using Azure Functions(v2) and blob storage. I want to generate a zip from many blob files.

The blob file is large and large in size so as to reach the Threshold memory of Functions.

I use System.IO.Compression.ZipArchive and I referred to How to zip huge files in a blob using stream

using (var blobStream = await archiveBlob.OpenWriteAsync()) using (var resultArchive = new ZipArchive(blobStream, ZipArchiveMode.Create, true)) ....

But next, I reached Threashold 5 minutes. So I try to split the function and gradually add files to Zip, but exception occurred at ZipArchive.

using (var resultArchive = new ZipArchive(blobStream, ZipArchiveMode.Update, true)) --> Microsoft.Azure.WebJobs.FunctionFailedException "Update mode requires a stream with read, write, and seek capabilities."

  • Can BlockBlob be opened with read and write capability?
  • Or have other ideas?

thanks.

CHEEKATLAPRADEEP
  • 12,191
  • 1
  • 19
  • 42
PUNIO
  • 45
  • 5
  • 1
    Why don't you just upload files as single blobs and store a reference to them as a a collection in a table somewhere. If someone wants that archive, you can still give them all their files. And please show us your error. When writing blobs, you have 10 minutes per megabyte before you run into a timeout. – Marco Nov 05 '18 at 06:43
  • Original files are already in blob.These are over 10000 files.So we are asked to download it one archive. The first error is an exception due to Threshold (5 min) of Functions.The following error is an exception of ZipArchive's constructor. – PUNIO Nov 06 '18 at 00:24
  • @PUNIO It seems your app is on consumption plan so there's 5 minutes limit by default. Try to extend function timeout threshold by adding ["functionTimeout": "00:10:00" in host.json](https://learn.microsoft.com/en-us/azure/azure-functions/functions-host-json#functiontimeout). – Jerry Liu Nov 06 '18 at 01:15
  • @JerryLiu I also thought about it.However, when it reached 5 min Threshold, it was able to process only about 2000 files. This is not enough even for the maximum 10 minutes.It is expensive to use "App Service plan", so I want to avoid it if possible. – PUNIO Nov 06 '18 at 05:25
  • you might have to look at Durable functions. https://learn.microsoft.com/en-us/azure/azure-functions/durable-functions-overview – Adithya Nov 06 '18 at 15:28
  • @AdithyaMorampudi I'm using Durable functions.But functions called from Orchestrator have Threshold as well. – PUNIO Nov 07 '18 at 00:32

1 Answers1

0

I have come to the same problem. Since nobody answered anything that would bring you closer to the solution, I will tell you my experience in mitigating the problem. You have 2 possible solutions

  1. Split the file into multiple zipped archives (optionally zip those already zipped archives into one at the end). You can achieve concurrency here by using Azure Durable Functions.

  2. Move zipping process to Azure WebJob. Unlike Azure Functions, Azure WebJobs do not have that kind of strict time limitation such as Azure Functions, and they are specifically made for this kind of long-running tasks. You can trigger WebJob from Azure Function or from your web api.

Tomov Nenad
  • 154
  • 5
  • It's a good idea to run it in WebJob. It's a pity that it is not completed with Functions. – PUNIO Sep 27 '21 at 00:58