Azure Blob Storage DownloadToStreamAsync hangs during network change

Question

I've been having an issue with the Microsoft.WindowsAzure.Storage v9.3.3 and Microsoft.Azure.Storage.Blob v11.1.0 NuGet libraries. Specifically when download a large file. If you change your network during the "DownloadToStreamAsync" method the call hangs. I've been seeing my code, which processes a lot of files, hang occasionally and I've been trying to narrow it down. I think the network change might be a reliable way of triggering some failure in the Azure Blob Storage Libraries.

More info about the issue;

When I unplug my network cable my computer switches to WiFi but the request never resumes
If I start the download on WiFi and then plug in my network cable the same error occurs
The “ServerTimeout” property never fails the request or acts as expected in accordance to the Documentation
The “MaximumExecutionTime” property does fail the request but we don’t want to limit ourselves to a certain time period, especially because we’re dealing with large files

The following code fails 100% of the time if the network is changed during the call.

static void Main(string[] args)
{
    try
    {
        CloudStorageAccount.TryParse("<Connection String>", out var storageAccount);
        var cloudBlobClient = storageAccount.CreateCloudBlobClient();
        var container = cloudBlobClient.GetContainerReference("<Container Reference>");
        var blobRef = container.GetBlockBlobReference("Large Text.txt");
        Stream memoryStream = new MemoryStream();
        BlobRequestOptions optionsWithRetryPolicy = new BlobRequestOptions() { ServerTimeout = TimeSpan.FromSeconds(5), RetryPolicy = new LinearRetry(TimeSpan.FromSeconds(20), 4) };
        blobRef.DownloadToStreamAsync(memoryStream, null, optionsWithRetryPolicy, null).GetAwaiter().GetResult();
        Console.WriteLine("Completed");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Exception: {ex.Message}");
    }
    finally
    {
        Console.WriteLine("Finished");
    }
}

I've found this active issue in the Azure Storage GitHub but it seems inactive.

Is there any other approach I could take to reliably and efficiently download a blob or something I'm missing when using this package?

Sorry, I've edited my post now. I wondered if there was anything else I could be doing? — Connor Dickson, Nov 05 '19 at 08:16

score 0 · Accepted Answer · answered Nov 05 '19 at 09:48

Thanks to Mohit for the suggestion.

Create a Task to check the stream length in the background
If the stream hasn't increased in a set period of time, cancel the DownloadToStreamAsync

DISCLAIMER: I haven't written tests around this code or how to make it run in a performant way as you couldn't have a wait like this for every file you process. I might need to cancel the initial task if the download completes, I don't know yet, I just wanted to get it working first. I don't deem it production ready.

// Create download cancellation token
var downloadCancellationTokenSource = new CancellationTokenSource();
var downloadCancellationToken = downloadCancellationTokenSource.Token;

var completedChecking = false;

// A background task to confirm the download is still progressing
Task.Run(() =>
{
    // Allow the download to start
    Task.Delay(TimeSpan.FromSeconds(2)).GetAwaiter().GetResult();

    long currentStreamLength = 0;
    var currentRetryCount = 0;
    var availableRetryCount = 5;

    // Keep the checking going during the duration of the Download
    while (!completedChecking)
    {
        Console.WriteLine("Checking");
        if (currentRetryCount == availableRetryCount)
        {
            Console.WriteLine($"RETRY WAS {availableRetryCount} - FAILING TASK");
            downloadCancellationTokenSource.Cancel();
            completedChecking = true;
        }

        if (currentStreamLength == memoryStream.Length)
        {
            currentRetryCount++;
            Console.WriteLine($"Length has not increased. Incremented Count: {currentRetryCount}");
            Task.Delay(TimeSpan.FromSeconds(10)).GetAwaiter().GetResult();
        }
        else
        {
            currentStreamLength = memoryStream.Length;
            Console.WriteLine($"Download in progress: {currentStreamLength}");
            currentRetryCount = 0;
            Task.Delay(TimeSpan.FromSeconds(1)).GetAwaiter().GetResult();
        }
    }
});

Console.WriteLine("Starting Download");

blobRef.DownloadToStreamAsync(memoryStream, downloadCancellationToken).GetAwaiter().GetResult();

Console.WriteLine("Completed Download");
completedChecking = true;

Console.WriteLine("Completed");

Azure Blob Storage DownloadToStreamAsync hangs during network change

1 Answers1