Polly: Retry request with StreamContent and MemoryStream - no content on second try

Question

I'm using Polly in combination with Microsoft.Extensions.Http.Polly to handle communication with an external API which has rate-limiting (N requests / second).I'm also using .NET 6.

The policy itself works fine for most requests, however it doesn't work properly for sending (stream) data. The API Client requires the usage of MemoryStream. When the Polly policy handles the requests and retries it, the stream data is not sent. I verified this behavior stems from .NET itself with this minimal example:

using var fileStream = File.OpenRead(@"C:\myfile.pdf");
using var memoryStream = new MemoryStream();
await fileStream.CopyToAsync(memoryStream);

var response = await httpClient.SendAsync(
    new HttpRequestMessage
    {
        // The endpoint will fail the request on the first request
        RequestUri = new Uri("https://localhost:7186/api/test"),
        Content = new StreamContent(memoryStream),
        Method = HttpMethod.Post
    }
);

Inspecting the request I see that Request.ContentLength is the length of the file on the first try. On the second try it's 0.

However if I change the example to use the FileStream directly it works:

using var fileStream = File.OpenRead(@"C:\myfile.pdf");

var response = await httpClient.SendAsync(
    new HttpRequestMessage
    {
        // The endpoint will fail the request on the first request
        RequestUri = new Uri("https://localhost:7186/api/test"),
        Content = new StreamContent(fileStream ),
        Method = HttpMethod.Post
    }
);

And this is my Polly policy that I add to the chain of AddHttpClient.

public static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
    return Policy
        .HandleResult<HttpResponseMessage>(response =>
        {
            return response.StatusCode == System.Net.HttpStatusCode.Forbidden;
        })
        .WaitAndRetryAsync(4, (retry) => TimeSpan.FromSeconds(1));
}

My question:

How do I properly retry requests where StreamContent with a stream of type MemoryStream is involved, similar to the behavior of FileStream?

Edit for clarification:

I'm using an external API Client library (Egnyte) which accepts an instance of HttpClient

public class EgnyteClient {
   public EgnyteClient(string apiKey, string domain, HttpClient? httpClient = null){
   ...
   }
}

I pass an instance which I injected via the HttpContextFactory pattern. This instance uses the retry policy from above.

This is my method for writing a file using EgnyteClient

public async Task UploadFile(string path, MemoryStream stream){
   // _egnyteClient is assigned in the constructor
   await _egnyteClient.Files.CreateOrUpdateFile(path, stream);
}

This method call works (doesn't throw an exception) even when the API sometimes returns a 403 statucode because the internal HttpClient uses the Polly retry policy. HOWEVER the data isn't always properly transferred since it just works if it was the first attempt.

Answer 1

The root cause of your problem could be the following: once you have sent out a request then the MemoryStream's Position is at the end of the stream. So, any further requests needs to rewind the stream to be able to copy it again into the StreamContent (memoryStream.Position = 0;).

---

Here is how you can do that with retry:

private StreamContent GetContent(MemoryStream ms)
{
   ms.Position = 0;
   return new StreamContent(ms);
}

var response = await httpClient.SendAsync(
    new HttpRequestMessage
    {
        RequestUri = new Uri("https://localhost:7186/api/test"),
        Content = GetContent(memoryStream),
        Method = HttpMethod.Post
    }
);

This ensures that the memoryStream has been rewinded for each each retry attempt.

---

UPDATE #1 After receiving some clarification and digging in the source code of the Egnyte I think I know understand the problem scope better.

• A 3rd party library receives an HttpClient instance which is decorated with a retry policy (related source code)
• A MemoryStream is passed to a library which is passed forward as a StreamContent as a part of an HttpRequestMessage (related source code)
• HRM is passed directly to the HttpClient and the response is wrapped into a ServiceResponse (related source code)

Based on the source code you can receive one of the followings:

• An HttpRequestException thrown by the HttpClient
• An EgnyteApiException or QPSLimitExceededException or RateLimitExceededException thrown by the ExceptionHelper
• An EgnyteApiException thrown by the SendRequestAsync if there was a problem related to the deserialization
• A ServiceResponse from SendRequestAsync

As far as I can see you can access the StatusCode only if you receive an HttpRequestException or an EgnyteApiException.

Because you can't rewind the MemoryStream whenever an HttpClient performs a retry I would suggest to decorate the UploadFile with retry. Inside the method you can always set the stream parameter's Position to 0.

public async Task UploadFile(string path, MemoryStream stream){
   stream.Position = 0;
   await _egnyteClient.Files.CreateOrUpdateFile(path, stream);
}

So rather than decorating the entire HttpClient you should decorate your UploadFile method with retry. Because of this you need to alter the policy definition to something like this:

public static IAsyncPolicy GetRetryPolicy()
    => Policy
        .Handle<EgnyteApiException>(ex => ex.StatusCode == HttpStatusCode.Forbidden)
        .Or<HttpRequestException>(ex => ex.StatusCode == HttpStatusCode.Forbidden)
        .WaitAndRetryAsync(4, _ => TimeSpan.FromSeconds(1));

Maybe the Or builder clause is not needed because I haven't seen any EnsureSuccessStatusCode call anywhere, but for safety I would build the policy like that.