I know about uploading in chunks, do we have to do something on receiving end?

Question

my azure function receives large video files and images and stores it in Azure blob. Client API is sending data in chunks to my Azure htttp trigger function. Do I have to do something at receiving-end to improve performance like receiving data in chunks?

Bruce, actually Client code is being developed by some other team. right now i am testing it by postman and getting files from multipart http request.

foreach (HttpContent ctnt in provider.Contents)  {

                var dataStream = await ctnt.ReadAsStreamAsync();
 if (ctnt.Headers.ContentDisposition.Name.Trim().Replace("\"", "") == "file")
               {                
                        byte[] ImageBytes = ReadFully(dataStream);
                        var fileName = WebUtility.UrlDecode(ctnt.Headers.ContentDisposition.FileName);                         

              } }

ReadFully Function

 public static byte[] ReadFully(Stream input){
using (MemoryStream ms = new MemoryStream())
{
    input.CopyTo(ms);
    return ms.ToArray();
}}

Could you share the client code snippet for uploading data in chunks, and the current code in your azure function for uploading the file to your blob storage? I assume that you could did some additional settings and upload the file to your blob storage in parallel within your azure function to improve the performance. — Bruce Chen, Feb 01 '18 at 09:09

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

As BlobRequestOptions.ParallelOperationThread states as follows:

Gets or sets the number of blocks that may be simultaneously uploaded.

Remarks:

When using the UploadFrom* methods on a blob, the blob will be broken up into blocks. Setting this value limits the number of outstanding I/O "put block" requests that the library will have in-flight at a given time. Default is 1 (no parallelism). Setting this value higher may result in faster blob uploads, depending on the network between the client and the Azure Storage service. If blobs are small (less than 256 MB), keeping this value equal to 1 is advised.

I assumed that you could explicitly set the ParallelOperationThreadCount for faster blob uploading.

var requestOption = new BlobRequestOptions()
{
    ParallelOperationThreadCount = 5 //Gets or sets the number of blocks that may be simultaneously uploaded.
};

//upload a blob from the local file system
await blockBlob.UploadFromFileAsync("{your-file-path}",null,requestOption,null);

//upload a blob from the stream
await blockBlob.UploadFromStreamAsync({stream-for-upload},null,requestOption,null);

foreach (HttpContent ctnt in provider.Contents)

Based on your code, I assumed that you retrieve the provider instance as follows:

MultipartMemoryStreamProvider provider = await request.Content.ReadAsMultipartAsync();

At this time, you could use the following code for uploading your new blob:

var blobname = ctnt.Headers.ContentDisposition.FileName.Trim('"');
CloudBlockBlob blockBlob = container.GetBlockBlobReference(blobname);
//set the content-type for the current blob
blockBlob.Properties.ContentType = ctnt.Headers.ContentType.MediaType;
await blockBlob.UploadFromStreamAsync(await ctnt.Content.ReadAsStreamAsync(), null,requestOption,null);

I would prefer use MultipartFormDataStreamProvider which would store the uploaded files from the client to the file system instead of MultipartMemoryStreamProvider which would use the server memory for temporarily storing the data sent from the client. For the MultipartFormDataStreamProvider approach, you could follow this similar issue. Moreover, I would prefer use the Azure Storage Client Library with my Azure function, you could follow Get started with Azure Blob storage using .NET.

UPDATE:

Moreover, you could follow this tutorial about breaking a large file into small chunks, upload them in the client side, then merge them back in your server side.

thanks Bruce.Is there anything that client is uploading in chunks and my function receive in chunks ,merge and save it as a file in blob? — Annie, Feb 02 '18 at 09:29
I may misunderstand your question before. For the Client API to send data in chunks, I assumed that you need to make your server-side to support this feature. Here is a tutorial talking about the similar scenario, you could follow [here](https://www.codeproject.com/Articles/1034347/Upload-large-files-to-MVC-WebAPI-using-partitionin) about the implementation. Moreover, you could still follow the part about speeding up the uploading your file to the blob storage on your server-side. — Bruce Chen, Feb 02 '18 at 09:43
The example you shared is for uploading data in chunks in client API...that other team will do. In function where I am receiving that data from request object, do I have to receive in chunks and merge those chunks? — Annie, Feb 02 '18 at 10:32
Per my understanding, the way for uploading data in chunks in the client depends on your server side code implementation. Your client needs to use the APIs expose by your server side for uploading file(s). I think you need to communicate with your team members whose handle the client side code. — Bruce Chen, Feb 05 '18 at 01:57

I know about uploading in chunks, do we have to do something on receiving end?

1 Answers1

Linked