Uploading Files more than 30 MB to Azure fileshare locaton from FTP using an console application(.net-C#)

Question

I'm using a C# (.net core) console program to upload a large file (.CSV) (limit 30 MB) to an Azure Fileshare location from an FTP site. In some cases, I'm getting incomplete writes in the Azure fileshare location. However, it works most of the time when I upload in chunks. Due to incomplete writing, the file might get damaged at the destination Fileshare location (missing a few rows and contents from the source location). The code I use is shown below. Please help me improve the procedure or provide other solutions.

public async void WriteFileToFileShare(ShareClient share, string azureFolder, string validDataFilePath, Stream stream)
        {
            ShareDirectoryClient directory = share.GetDirectoryClient(azureFolder);

            //  Azure allows for 4MB max uploads  (4 x 1024 x 1024 = 4194304)
            const int uploadLimit = 4194304;
            stream.Seek(0, SeekOrigin.Begin);   // ensure stream is at the beginning
            var fileClient = await directory.CreateFileAsync(Path.GetFileName(validDataFilePath), stream.Length);
            // If stream is below the limit upload directly
            if (stream.Length <= uploadLimit)
            {
                await fileClient.Value.UploadRangeAsync(new HttpRange(0, stream.Length), stream);
                return;
            }
            int bytesRead;
            long index = 0;
            byte[] buffer = new byte[uploadLimit];
            // Stream is larger than the limit so we need to upload in chunks
            while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
            {
                // Create a memory stream for the buffer to upload
                using MemoryStream ms = new MemoryStream(buffer, 0, bytesRead);
                await fileClient.Value.UploadRangeAsync(ShareFileRangeWriteType.Update, new HttpRange(index, ms.Length), ms);
                index += ms.Length; // increment the index to the account for bytes already written
            }
        }

increase uploadLimit value = * 1024 * 1024. E.g. 50 * 1024 * 1024 for 50MB https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-scalable-app-upload-files?tabs=dotnet — Nayan, May 25 '22 at 15:42
4MB is the maximum we can upload in one chunk. that's the problem — Nair, May 27 '22 at 13:46
Azure File Sync Agent can be used to Maintain content sync between on-premise and Azure File Share. https://learn.microsoft.com/en-us/azure/storage/file-sync/file-sync-deployment-guide?tabs=azure-portal%2Cproactive-portal — Nayan, May 27 '22 at 13:50

score 0 · Answer 1 · answered Jun 01 '22 at 06:47

As suggested by @Nayan, To upload more then 30 mb to an Azure Fileshare location from an FTP site .

Thank you for your valuable suggestion ,posting it as an answer for other community members who have similar issue can find and fix their problem.

The Azure File Sync Agent can keep content synchronised between on-premises and Azure File Share.

Here is the example of code based on the MS DOC:-

private static async Task UploadFilesAsync()
{
    // Create five randomly named containers to store the uploaded files.
    BlobContainerClient[] containers = await GetRandomContainersAsync();

    // Path to the directory to upload
    string uploadPath = Directory.GetCurrentDirectory() + "\\upload";

    // Start a timer to measure how long it takes to upload all the files.
    Stopwatch timer = Stopwatch.StartNew();

    try
    {
        Console.WriteLine($"Iterating in directory: {uploadPath}");
        int count = 0;

        Console.WriteLine($"Found {Directory.GetFiles(uploadPath).Length} file(s)");

        // Specify the StorageTransferOptions
        BlobUploadOptions options = new BlobUploadOptions
        {
            TransferOptions = new StorageTransferOptions
            {
                // Set the maximum number of workers that 
                // may be used in a parallel transfer.
                MaximumConcurrency = 8,

                // Set the maximum length of a transfer to 50MB.
                MaximumTransferSize = 50 * 1024 * 1024
            }
        };

        // Create a queue of tasks that will each upload one file.
        var tasks = new Queue<Task<Response<BlobContentInfo>>>();

        // Iterate through the files
        foreach (string filePath in Directory.GetFiles(uploadPath))
        {
            BlobContainerClient container = containers[count % 5];
            string fileName = Path.GetFileName(filePath);
            Console.WriteLine($"Uploading {fileName} to container {container.Name}");
            BlobClient blob = container.GetBlobClient(fileName);

            // Add the upload task to the queue
            tasks.Enqueue(blob.UploadAsync(filePath, options));
            count++;
        }

        // Run all the tasks asynchronously.
        await Task.WhenAll(tasks);

        timer.Stop();
        Console.WriteLine($"Uploaded {count} files in {timer.Elapsed.TotalSeconds} seconds");
    }
    catch (RequestFailedException ex)
    {
        Console.WriteLine($"Azure request failed: {ex.Message}");
    }
    catch (DirectoryNotFoundException ex)
    {
        Console.WriteLine($"Error parsing files in the directory: {ex.Message}");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Exception: {ex.Message}");
    }
}

For more information please refer the below links:-

SO THREAD| Uploading to Azure File Storage fails with large files.

& How much large file we can upload to Azure file share in a single shot in .net core 3.1

Uploading Files more than 30 MB to Azure fileshare locaton from FTP using an console application(.net-C#)

1 Answers1