Using Polly in C#, can I wait until a timespan elapses OR a task terminates before retrying?

Question

I am trying to use the Polly package in C#. I want to run some code and then, if it fails, wait and retry. Currently my loop looks similar to this:

var successful = false
while (!successful){
    // Try to perform operation.
    successful = TryToDoStuff()
    if (!successful){
        // Wait, then retry.
        await Task.WhenAny(
            taskCompletionSource1.Task,
            taskCompletionSource1.Task,
            Task.Delay(TimeSpan.FromSeconds(10)));
    }
}

I.e.: Wait 10 seconds OR until one of these task completion sources gets a signal and terminates. Then retry.

What I want to do is something like this (which is not supported by the Polly API):

Policy
    .Handle<RetryException>()
    .WaitAndRetryForever(
        Task.WhenAny(
            taskCompletionSource1.Task,
            taskCompletionSource1.Task,
            Task.Delay(TimeSpan.FromSeconds(10))))
    .Execute(TryToDoStuff); // Method TryToDoStuff will throw RetryException if it fails

Is it possible to do something like this with Polly? Can I wait for anything other than a TimeSpan?

Regarding the two tasks I await in the above example: One task is a cancellation indicating that the entire thing should shut down. The other is a "wake up for connection attempt" task whose termination indicates that "this object's state has changed; try to call it again". In both cases I want my loop to continue to the next iteration immediately instead of waiting for the timeout to elapse.

Currently waiting for the timeout is not so bad since it's only 10 seconds, but if I change it to exponential backoff, then suddenly the timeout can be very long. Hence my desire to interrupt the timeout and proceed straight to the next iteration.

Note: It is not imperative that my retrying loop follow the async-await pattern. It is OK if the waiting part is synchronous and blocking. I just want to be able to cancel the wait using a task completion source.

I need a little more explanation in order to help. Based on my understanding you need to do two things: setup a retry logic to handle failures and set a timeout for the operation that you want to try. Is this correct ? — Enrico Massone, Dec 18 '19 at 13:45
I'm asking because excecuting an operation by adding a timeout and setting up a retry logic of some sort are very different tasks. So we need to understand exactly what you want to achieve (from a functional point of view) — Enrico Massone, Dec 18 '19 at 13:50
You're right; my code example was too vague and unclear. I have expanded it. Is it clearer now? — Claus Appel, Dec 18 '19 at 14:05
Good now it's clearer. I don't fully understand the wait logic. You are basically saying that there are 3 events that can possibly signal the end of the waiting. First of all a 10 seconds timeout, but there are also two tasks representing other operations. Are those other tasks really needed in your wait and retry logic ? What do they stand for ? — Enrico Massone, Dec 18 '19 at 14:09
One task is a cancellation indicating that the entire thing should shut down. The other is a "wake up for connection attempt" task whose termination indicates that "this object's state has changed; try to call it again". In both cases I want my loop to continue to the next iteration immediately instead of waiting for the timeout to elapse. (Currently waiting for the timeout is not so bad since it's only 10 seconds, but if I change it to exponential backoff, then suddenly the timeout can be very long.) — Claus Appel, Dec 18 '19 at 14:19
Based on my understanding you cant do that simply with polly. Because if you use one of the polly retry policy you can only express the waiting time as a timespan (based on my understanding of polly). I see two solution here. The first one is avoiding polly and do a poor man implementation via loop similarly to what you did above. The second one is trying to reshape the entire thing — Enrico Massone, Dec 18 '19 at 20:50
For instance above you said that one of the taskis a cancellation indicating that the "entire thing" must shut down. You can do this simply by passing a cancellation token and ensuring that your implementation listens to the token and that, at the same time, someone else is able to ask the token to cancel itself under the proper condition. This is way simpler and can be done using any code written with cancellation support (and polly policies supports cancellation) — Enrico Massone, Dec 18 '19 at 20:53
What about the wake up task ? Is it really necessary for you to interrupt the waiting between two attempts as soon as that task completes ? — Enrico Massone, Dec 18 '19 at 20:54
There is probably a better pattern. You can refuse any incoming request until your initialization tasks completes successfully. I give you an example in the context of web applications. Sonetimes in a web application a task needed to bootstrap the app exists. A good pattern to handle this scenario is letting the web application to start and once started start trying to complete the bootstrap task (whatever this means). — Enrico Massone, Dec 18 '19 at 20:57
While the bootstrap task is still in progress the web application returns a 503 service unavailable to all the incoming requests and this lasts until the bootstrap task completes. Once this happens the web app can start serve requests as usual. Can you apply a similar pattern in your scenario ? — Enrico Massone, Dec 18 '19 at 20:59

mountain traveller · Accepted Answer · 2019-12-19T09:52:18.353

All Polly policies and executions can respond to CancellationTokens to signal cancellation.

If I understand correctly, there are two requirements:

Retry immediately (no further delay) if a cancellation is signalled
Retry with time-delay (10-second or exponential) if a RetryException occurs

You can express this with a Polly policy for each:

var retryImmediatelyOnCancellation = Policy
    .Handle<OperationCanceledException>()
    .RetryForever();

var retryWithDelay = Policy
    .Handle<RetryException>()
    .WaitAndRetryForever(/* specify your desired delay or delay sequence for retries */);

and then execute through the two retry policies in a nested fashion.

Something like:

retryImmediatelyOnCancellation.Execute(() => 
{
    CancellationTokenSource externalCancellation = ... // Get the CancellationTokenSource signalling external cancellation.
    CancellationTokenSource wakeUpConnectionAttemptCancellation = ... // Get the CancellationTokenSource signalling "wake up connection attempt".
    CancellationTokenSource combinedCancellationSource = CancellationTokenSource.CreateLinkedTokenSource(externalCancellation.Token, wakeUpConnectionAttemptCancellation.Token);

    retryWithDelay.Execute(ct => TryDoStuff(), combinedCancellationSource.Token);
});

The delay made by the retryWithDelay policy cancels immediately if the CancellationToken is signalled.

CancellationToken is a more natural cancellation signal than TaskCompletionSource. However, if you need (for whatever reason) to stick with using a TaskCompletionSource as the signal of those events, you can convert that to cancel a CancellationTokenSource with something simple like:

taskCompletionSource.Task.ContinueWith(t => cancellationTokenSource.Cancel());

Note that both TaskCompletionSource and CancellationToken are single-use only: once completed or cancelled, they cannot be reset (uncompleted or uncancelled). The code sample above moves creation of the CancellationTokenSources inside the retry-on-cancellation loop in order that you do get fresh CancellationTokenSources after each cancellation signal.

All of this also works with async Polly policies if you switch to async.

score 0 · Answer 2 · answered Dec 19 '19 at 08:02

The best solution I could come up with was this:

var successful = false
while (!successful){

    // Create cancellation token that gets cancelled when one of the tasks terminates.
    var cts = new CancellationTokenSource();
    _ = Task.Run(async () =>
    {
        await Task.WhenAny(
            taskCompletionSource1.Task,
            taskCompletionSource1.Task);
        cts.Cancel();
    });

    // Try to perform operation.
    Policy
        .Handle<RetryException>()
        .WaitAndRetryForever(
            TimeSpan.FromSeconds(10))
        .Execute(
            // Method TryToDoStuff will throw RetryException if it fails 
            ct => TryToDoStuff(), 
            // Pass in cancellation token.
            cts.Token);
}

This seems to work. But I might end up doing this without Polly.

Using Polly in C#, can I wait until a timespan elapses OR a task terminates before retrying?

2 Answers2