Ensure Polly Policy Runs at Least Once

Question

So, I'm writing some retry logic for acquiring a lock using Polly. The overall timeout value will be provided by the API caller. I know I can wrap a policy in an overall timeout. However, if the supplied timeout value is too low is there a way I can ensure that the policy is executed at least once?

Obviously I could call the delegate separately before the policy is executed but I was just wondering if there was a way to express this requriment in Polly.

var result = Policy.Timeout(timeoutFromApiCaller)
                    .Wrap(Policy.HandleResult(false)
                                .WaitAndRetryForever(_ => TimeSpan.FromMilliseconds(500))
                    .Execute(() => this.TryEnterLock());

If timeoutFromApiCaller is say 1 tick and there's a good chance it takes longer than that to reach the timeout policy then the delegate wouldn't get called (the policy would timeout and throw TimeoutRejectedException).

What I'd like to happen can be expressed as:

var result = this.TryEnterLock();

if (!result)
{
    result = Policy.Timeout(timeoutFromApiCaller)
                   .Wrap(Policy.HandleResult(false)
                               .WaitAndRetryForever(_ => TimeSpan.FromMilliseconds(500))
                   .Execute(() => this.TryEnterLock());
}

But it'd be really nice if it could be expressed in pure-Polly...

Currently your implementation does not have a global (overall) timeout. It operates as "local". Let's suppose that timeout is 2sec, retry count is 3 and retry delay is 4sec. It waits 2 seconds then it pauses 4 seconds before the next retry... It is important in which order you wrap your policies: if inner fails then it escalates to the outer. Local timeout: Retry > Timeout, Global Timeout: Timeout > Retry, Local+Overall: Timeout > Retry > Timeout. — Peter Csala, Jun 16 '20 at 11:25
Thanks @PeterCsala, would you know if what I'm trying to achieve is possible? i.e. always try once and then respect any *global* timeout? — Nick, Jun 16 '20 at 12:44
@Nick The truth is that out of the box you got the at least one call guarantee. Because retry count 3 means 4 attempts.The initial call is the first attempt. If it fails then the retry will be triggered. So your first retry will be your second attempt. — Peter Csala, Jun 16 '20 at 13:03
@PeterCsala, I've modified the code slightly so it actually represents what I'm trying to acheive. I.e. I have a timeout in the outerscope and retries in the inner scope. This will throw `TimeoutRejectedException` if the outer times out before the inner gets a chance to operate. I've also indicated how I'd like it to operate. — Nick, Jun 16 '20 at 15:00

Peter Csala · Accepted Answer · 2020-06-19T07:36:36.347

To be honest I don't understand what does it mean 1 tick, in your case? Is it a nanosecond or greater than that? Your global timeout should be greater than your local timeout.

But as I can see you have not specified a local one. TryEnterLock should receive a TimeSpan in order to do not block the caller for infinite time. If you look at the built in sync primitives most of them provide such a capabilities: Monitor.TryEnter, SpinLock.TryEnter, WaitHandle.WaitOne, etc.

So, just to wrap it up:

var timeoutPolicy = Policy.Timeout(TimeSpan.FromMilliseconds(1000));
var retryPolicy = Policy.HandleResult(false)
       .WaitAndRetryForever(_ => TimeSpan.FromMilliseconds(500));
var resilientStrategy = Policy.Wrap(timeoutPolicy, retryPolicy);    

var result = resilientStrategy.Execute(() => this.TryEnterLock(TimeSpan.FromMilliseconds(100)));

The timeout and delay values should be adjusted to your business needs. I highly encourage you to log when the global Timeout (onTimeout / onTimeoutAsync) fires and when the retries (onRetry / onRetryAsync) to be able to fine tune / calibrate these values.

EDIT: Based on the comments of this post

As it turned out there is no control over the timeoutFromApiCaller so it can be arbitrary small. (In the given example it is just a few nano-seconds, with the intent to emphasize the problem.) So, in order to have at least one call guarantee we have to make use the Fallback policy.

Instead of calling manually upfront the TryEnterLock outside the policies, we should call it as the last action to satisfy the requirement. Because policies uses escalation, that's why whenever the inner fails then it delegates the problem to the next outer policy.

So, if the provided timeout is so tiny that action can not finish until that period then it will throw a TimeoutRejectedException. With the Fallback we can handle that and the action can be performed again but now without any timeout constraint. This will provide us the desired at least one guarantee.

var atLeastOnce = Policy.Handle<TimeoutRejectedException>
    .Fallback((ct) => this.TryEnterLock());
var globalTimeout = Policy.Timeout(TimeSpan.FromMilliseconds(1000));
var foreverRetry = Policy.HandleResult(false)
       .WaitAndRetryForever(_ => TimeSpan.FromMilliseconds(500));

var resilientStrategy = Policy.Wrap(atLeastOnce, globalTimeout, foreverRetry);    
var result = resilientStrategy.Execute(() => this.TryEnterLock());

Thanks Peter. The `TryEnterLock` is an existing method which calls out to a SPROC to write a value to a db table - basically doing row-level locking without using row-level locking! (don't ask me!). I'm trying to implement a simple wait/retry/timeout as currently if we don't get the lock on the first attempt we just fail. — Nick, Jun 17 '20 at 10:20
My concern with using Polly for this is that there is chance we never call `TryEnterLock` if the `Timeout` times out (i.e. API caller specified a silly timeout value or something slows things down before the first invocation of the delegate. So I'd like to express a wait/retry/timeout but which is *guaranteed* to execute at least once. — Nick, Jun 17 '20 at 10:20
@Nick 1 tick around 100 nano-seconds. Are you sure that is a valid timeout in your use case where you want to make a db call? I would consider to enforce some validation logic against `timeoutFromApiCaller`. If you are using the deadline / distributed timeout pattern then you can fail fast if the remaining timeout is under a reliable threshold. — Peter Csala, Jun 17 '20 at 10:38
Yeah, I'm aware of all that and I'm just using 100ns to prove a point. I'm just concerned with determinism, i.e. I want to be able to guarantee that no matter what the timeouts Polly is always going to call my delegate at least once. Now, I can do that by always calling my delegate first before executing the policy if required however it'd be nice if Polly had a natural way to express this. — Nick, Jun 18 '20 at 05:08
@Nick You can do that by using the **Fallback policy**. Instead of calling it directly (outside of the policies) at the first place, you can do that as a last chance. If the call wrapped inside the policies has thrown a `TimeoutRejectedException` then you can call the method itself. So, your resilient strategy would look like this: Fallback >> Retry >> Timeout >> Action. Does it make sense to you? — Peter Csala, Jun 18 '20 at 05:59
Ah, yes that sounds like it. If you update your answer with this, I'll accept it. — Nick, Jun 19 '20 at 06:18

Ensure Polly Policy Runs at Least Once

1 Answers1