Polly Timeout policy wrapped in WaitAndRetry policy does not work

Question

I have a service which writes files over some network using FileStream, without a write timeout. I added a Polly WaitAndRetry policy to handle occasional write failures, and that works great. I noticed that sometimes a write just hangs though, so I'm trying to wrap a Timeout policy inside the retry policy in order to limit each write attempt.

public class Retrier
{
    private readonly CancellationToken _cancellationToken;

    public Retrier(CancellationToken cancellationToken)
    {
        _cancellationToken = cancellationToken;
    }

    public void Start(IService service)
    {
        var retryPolicy = Policy
            .Handle<IOException>()
            .WaitAndRetry(
                retryCount: 3,
                sleepDurationProvider: (retries) => TimeSpan.FromSeconds(retries * 10));
        var timeoutPolicy = Policy.Timeout(seconds: 60);
        var combinedPolicy = retryPolicy.Wrap(timeoutPolicy);
        var result = combinedPolicy.ExecuteAndCapture(
            (ct) => 
            {
                service.Write(ct);
            },
            _cancellationToken);

        // Do some other stuff
    }
}

However, with the wrapped policy, the Write action does not get called at all, e.g. ExecuteAndCapture is entered and action called, but execution just continues to "do other other stuff" below. A simple test method verifies that Write is called zero times.

[TestMethod]
public void Retrier_Start_When_IOExceptionOnce_CallsExecuteTwice()
{
    var attempt = 0;
    var mock = new Mock<IService>();

    mock
        .Setup(svc => svc.Write(_cancellationTokenSource.Token))
        .Returns(() =>
        {
            attempt++; 

            if (attempt == 1)
            {
                throw new IOException("Failed");
            }

            return 1;
        })
        .Verifiable();

    _testee.Start(mock.Object);
    mock.Verify(svc => svc.Write(_cancellationTokenSource.Token), Times.Exactly(2));
}

Write is a simple service, no magic:

public int Write(CancellationToken cancellationToken)
{
    cancellationToken.ThrowIfCancellationRequested();

    var tempFullPath = ...

    using (var fw = new FileStream(tempFullPath, FileMode.Create, FileAccess.Write))
    {
        fw.Write(_buffer, 0, _buffer.Length);
    }

    File.Move(tempFullPath, _finalFullPath);

    return ++Count;
}

I went over similar questions, but couldn't find a solution. What am I doing wrong?

Could you please share with us the implementation of `service.Write`? — Peter Csala, Apr 19 '23 at 12:40
You have to call `ThrowIfCancellationRequested` not just at the very beginning of the method rather than all the pivot points. Like before write, before move. Please be aware if the cancellation is called after write then you have to do a cleanup — Peter Csala, Apr 19 '23 at 14:25

Peter Csala · Answer 1 · 2023-05-26T19:29:02.743

0

If the Timeout policy kicks in then it throws a TimeoutRejectedException.

So, you need to modify your retry to handle that exception as well.

var timeoutPolicy = Policy.Timeout(seconds: 60);
var retryPolicy = Policy
            .Handle<IOException>()
            .Or<TimeoutRejectedException>()
            .WaitAndRetry(
                retryCount: 3,
                sleepDurationProvider: (retries) => TimeSpan.FromSeconds(retries * 10));

var combinedPolicy = Policy.Wrap(retryPolicy, timeoutPolicy);

UPDATE #1

Here is a dotnet fiddle link with a working example: http://dotnetfiddle.net/uTvCJA

edited May 26 '23 at 19:29

answered Apr 19 '23 at 10:57

Peter Csala

17,736
16
35
75

Thanks for the answer Peter. I did suspect that at some point and tried exactly what you suggest, but behavior remained the same, so it's something else. – Dondey Apr 19 '23 at 11:36
@Dondey Could you please try to change `ExecuteAndCapture` to `Execute` and see what Exception is thrown. – Peter Csala Apr 19 '23 at 11:50
No exception is thrown: - Capture returns success and no exception in the result object. - Adding Or catches nothing. - Wrapping the entire thing in a try-catch catches nothing. - All the above hold whether ExecuteAndCapture or Execute. Additionally, an onTimeout handler in the Timeout policy is never triggered. This is truly baffling. – Dondey Apr 19 '23 at 12:03
@Dondey Can you please two more cases? 1) Can you explicitly set the `TimeoutStrategy` to `Optimistic`? 2) Can you set the cancellationToken to `CancellationToken.None`? – Peter Csala Apr 19 '23 at 12:12
Explicitly setting to Optimistic + CancellationToken.None didn't help either – Dondey Apr 19 '23 at 12:19
@Dondey Dummy question but does the `service.Write` check the cancellationToken's state (either via `IsCancellationRequested` or `ThrowIfCancellationRequested`)? – Peter Csala Apr 19 '23 at 12:25
yes of course, ```cancellationToken.ThrowIfCancellationRequested()``` is the first line in service.Write – Dondey Apr 19 '23 at 12:26
@Dondey But not inside the mock. So, whenever you have stated none of the above helped then you checked it against the real code or against the unit test? – Peter Csala Apr 19 '23 at 12:31
@Dondey I've put together this dotnet fiddle example: https://dotnetfiddle.net/uTvCJA. Is there something which is missing from this from your example perspective? – Peter Csala Apr 19 '23 at 12:39
@Csela thas's a good point about the CancellationToken in the mock! But it didn't help either though. Your fiddle seems right, but when I apply the same on my code I don't see any improvement. Sad face. – Dondey Apr 19 '23 at 13:33
@Dondey It seems like there must be something else which is not shared with us :( – Peter Csala Apr 19 '23 at 13:54

Polly Timeout policy wrapped in WaitAndRetry policy does not work

1 Answers1