AsyncInterceptor with Polly (AOP retry style)

Question

For some part of my system I need to add retry logic for reading from the database. I have a number of repositories with async and sync read methods that I can't change. I found a simple solution - interception of all read methods with AsyncInterceptor and add retry read policy with Polly when database exception caught. Polly retries reading with some intervals.

Interceptor code:

public class RetriableReadAsyncInterceptor : IAsyncInterceptor
{
    public void InterceptSynchronous(IInvocation invocation)
    {
        invocation.ReturnValue = InternalInterceptSync(invocation);
    }

    public void InterceptAsynchronous(IInvocation invocation)
    {
        throw new NotImplementedException();
    }

    public void InterceptAsynchronous<TResult>(IInvocation invocation)
    {
        invocation.ReturnValue = InternalInterceptAsync<TResult>(invocation);
    }

    private IEnumerable<TimeSpan> RetryIntervals =>
        new[]
        {
            TimeSpan.FromSeconds(1),
            TimeSpan.FromSeconds(5),
            TimeSpan.FromSeconds(10),
            TimeSpan.FromSeconds(15)
        };

    private object InternalInterceptSync(IInvocation invocation)
    {
        return Policy
            .Handle<DatabaseException>()
            .WaitAndRetry(RetryIntervals, (exception, timeSpan) =>
            {
                Console.WriteLine($"Exception {timeSpan}");
            })
            .Execute(() =>
            {
                invocation.Proceed();
                return invocation.ReturnValue;
            });
    }

    private async Task<TResult> InternalInterceptAsync<TResult>(IInvocation invocation)
    {
        return await Policy
            .Handle<DatabaseException>()
            .WaitAndRetryAsync(RetryIntervals, (exception, timeSpan) =>
            {
                Console.WriteLine($"Exception {timeSpan}");
            })
            .ExecuteAsync(async () =>
            {
                invocation.Proceed();
                var task = (Task<TResult>)invocation.ReturnValue;
                return await task;
            });
    }
}

Repository code:

public class Repository : IRepository
{
    private int _exceptionsCoutner;

    public Entity GetById(int id)
    {
        if (_exceptionsCoutner <= 2)
        {
            _exceptionsCoutner++;
            throw new DatabaseException();
        }

        //read from db
        return new Entity {Id = id};
    }

    public async Task<Entity> GetByIdAsync(int id)
    {
        if (_exceptionsCoutner <= 2)
        {
            _exceptionsCoutner++;
            throw new DatabaseException();
        }

        //read from db
        return await Task.FromResult(new Entity { Id = id });
    }
}

Sync version of GetById works as expected (retries with intervals):

Exception 00:00:01
Exception 00:00:05
Exception 00:00:10

Async version of GetById retries but not waits for time interval elapsed:

Exception 00:00:01
Exception 00:00:01
Exception 00:00:01

I can't understand where is the problem. If you have any thoughts - please share. Full example can be found here.

For completeness for those coming to this question later: I provided a diagnosis of the problem (re-entrancy presumably caused by AsyncInterceptor) when [answering the same q on Polly github](https://github.com/App-vNext/Polly/issues/619#issuecomment-474529178) — mountain traveller, Mar 19 '19 at 19:09

score 1 · Accepted Answer · answered Oct 14 '19 at 09:23

It was a kind of 'chicken and egg' problem which can be solved now with newer version of Castle.Core (I tried version 4.4.0) leveraging the invocation.CaptureProceedInfo method:

private Task<TResult> InternalInterceptAsync<TResult>(IInvocation invocation)
{
    var capture = invocation.CaptureProceedInfo();

    return Policy
        .Handle<DatabaseException>()
        .WaitAndRetryAsync(RetryIntervals, (exception, timeSpan) =>
        {
            Console.WriteLine($"Exception {timeSpan}");
        })
        .ExecuteAsync(async () =>
        {
            capture.Invoke();
            var task = (Task<TResult>)invocation.ReturnValue;
            return await task;
        });
}

score 0 · Answer 2 · answered Mar 19 '19 at 13:58

OK, here is my naive implementation of retry:

public class Retry
{
    public static async Task<TResult> DoAsync<TResult, TException>(
        Func<Task<TResult>> action,
        TimeSpan retryInterval,
        int maxAttemptCount = 3)
        where TException : Exception
    {
        TException exception = null;
        var startDateTime = DateTime.UtcNow;

        for (var attempted = 0; attempted < maxAttemptCount; attempted++)
        {
            try
            {
                return await action().ConfigureAwait(false);
            }
            catch (TException ex)
            {
                exception = ex;
                Console.WriteLine($"Exception {DateTime.UtcNow - startDateTime}");
                await Task.Delay(retryInterval); //doesnt work
                //Thread.Sleep(retryInterval); works!
            }
        }

        throw exception;
    }
}

And interceptor:

private async Task<TResult> InternalInterceptAsync<TResult>(IInvocation invocation)
{
    return await Retry.DoAsync<TResult, DatabaseException>(async () =>
        {
            invocation.Proceed();
            var task = (Task<TResult>) invocation.ReturnValue;
            return await task.ConfigureAwait(false);
        },
        TimeSpan.FromSeconds(3),
        4);
}

Implementation with blocking Tread.Sleep works well, but with Task.Delay dont.

AsyncInterceptor with Polly (AOP retry style)

2 Answers2