Using Polly to do reconnects and timeouts

Question

I'm having a problem using Polly while trying to accomplish the following:

• Reconnect logic - I tried to create a Polly policy which works when you try to execute StartAsync without Internet connection. However, when it reaches ReceiveLoop, the policy has no longer impact over that method and if our connection stops at that point, it never tries to reconnect back. It simply throws the following exception: Disconnected: The remote party closed the WebSocket connection without completing the close handshake.. Perhaps I should have two policies: one in StartAsync and one in ReceiveLoop, but for some reason it doesn't feel right to me, so that's why I ask the question.

• Timeouts - I want to add timeouts for each ClientWebSocket method call e.g. ConnectAsync, SendAsync, etc. I'm not so familiar with Polly but I believe this policy automatically does that for us. However, I need someone to confirm that. By timeout, I mean similar logic to _webSocket.ConnectAsync(_url, CancellationToken.None).TimeoutAfter(timeoutMilliseconds), TimeoutAfter implementation can be found here. An example how other repos did it can be found here.

Simplified, I want to make this class resilient, which means instead of trying to connect to a dead web socket server for 30 seconds without success, no matter what the reason is, it should fail fast -> retry in 10 seconds -> fail fast -> retry again and so on. This wait and retry logic should be repeated until we call StopAsync or dispose the instance.

You can find the WebSocketDuplexPipe class on GitHub.

public sealed class Client : IDisposable
{
    private const int RetrySeconds = 10;
    private readonly WebSocketDuplexPipe _webSocketPipe;
    private readonly string _url;

    public Client(string url)
    {
        _url = url;
        _webSocketPipe = new WebSocketDuplexPipe();
    }

    public Task StartAsync(CancellationToken cancellationToken = default)
    {
        var retryPolicy = Policy
            .Handle<Exception>(e => !cancellationToken.IsCancellationRequested)
            .WaitAndRetryForeverAsync(_ => TimeSpan.FromSeconds(RetrySeconds),
                (exception, calculatedWaitDuration) =>
                {
                    Console.WriteLine($"{exception.Message}. Retry in {calculatedWaitDuration.TotalSeconds} seconds.");
                });

        return retryPolicy.ExecuteAsync(async () =>
        {
            await _webSocketPipe.StartAsync(_url, cancellationToken).ConfigureAwait(false);
            _ = ReceiveLoop();
        });
    }

    public Task StopAsync()
    {
        return _webSocketPipe.StopAsync();
    }

    public async Task SendAsync(string data, CancellationToken cancellationToken = default)
    {
        var encoded = Encoding.UTF8.GetBytes(data);
        var bufferSend = new ArraySegment<byte>(encoded, 0, encoded.Length);
        await _webSocketPipe.Output.WriteAsync(bufferSend, cancellationToken).ConfigureAwait(false);
    }

    private async Task ReceiveLoop()
    {
        var input = _webSocketPipe.Input;

        try
        {
            while (true)
            {
                var result = await input.ReadAsync().ConfigureAwait(false);
                var buffer = result.Buffer;

                try
                {
                    if (result.IsCanceled)
                    {
                        break;
                    }

                    if (!buffer.IsEmpty)
                    {
                        while (MessageParser.TryParse(ref buffer, out var payload))
                        {
                            var message = Encoding.UTF8.GetString(payload);

                            _messageReceivedSubject.OnNext(message);
                        }
                    }

                    if (result.IsCompleted)
                    {
                        break;
                    }
                }
                finally
                {
                    input.AdvanceTo(buffer.Start, buffer.End);
                }
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Disconnected: {ex.Message}");
        }
    }
}

Answer 1

Let me capture in an answer the essence of our conversation via comments.

ReceiveLoop with retry

Your retry policy will exit with success from the ExecuteAsync while you are waiting for the input.ReadAsync to finish. The reason is that you are not awaiting the ReceiveLoop rather you just kick it off in a fire and forget manner.

In other words, your retry logic will only apply for the StartAsync and the code before the await inside the ReceiveLoop.

The fix is to move the retry logic inside ReceiveLoop.

Timeout

Polly's Timeout policy can use either optimistic or pessimistic strategy. The former one heavily relies on the CancellationToken.

• So, if you pass for example CancellationToken.None to the ExecuteAsync then you basically says let TimeoutPolicy handle cancellation process.
• If you pass an already existing token then the decorated Task can be cancelled by the TimeoutPolicy or by the provided token.

Please bear in mind that it will throw TimeoutRejectedException not OperationCanceledException.

onTimeoutAsync

TimeoutAsync has several overloads which can accept one of the two onTimeoutAsync delegates

Func<Context, TimeSpan, Task, Task> onTimeoutAsync

or

Func<Context, TimeSpan, Task, Exception, Task> onTimeoutAsync

That can be useful to log the fact the timeout has occurred if you have an outer policy (for example a retry) which triggers on the TimeoutRejectedException.

Chaining policies

I suggest to use the Policy.WrapAsync static method instead of the WrapAsync instance method of the AsyncPolicy.

var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromMilliseconds(timeoutMs), TimeoutStrategy.Optimistic,
    (context, timeSpan, task, ex) =>
    {
        Console.WriteLine($"Timeout {timeSpan.TotalSeconds} seconds");
        return Task.CompletedTask;
    });

var retryPolicy = Policy
    .Handle<Exception>(ex =>
    {
        Console.WriteLine($"Exception tralal: {ex.Message}");
        return true;
    })
    .WaitAndRetryForeverAsync(_ => TimeSpan.FromMilliseconds(retryBackOffMs),
    (ex, retryCount, calculatedWaitDuration) =>
    {
        Console.WriteLine(
            $"Retrying in {calculatedWaitDuration.TotalSeconds} seconds (Reason: {ex.Message}) (Retry count: {retryCount})");
    });

var resilientStrategy = Policy.WrapAsync(retryPolicy, timeoutPolicy);

With this approach your retry policy's definition does not refer to the timeout policy explicitly. Rather you have two separate policies and a chained one.