What kind of exceptions give retries in Azure Service Bus SubscriptionClient?

Question

I am struggling with messages ending up in dead letter queue too quickly. I have specified an ExponentialRetry policy like this:

        private readonly RetryExponential _retryPolicy = new RetryExponential(
            TimeSpan.FromSeconds(1),
            TimeSpan.FromMinutes(20),
            10);

When SQL Server is temporarily down ("because its replica role is RESOLVING which does not allow connections. Try the operation again later.") then the messages end up in dead letter queue without retries.

What do I do to get retries?

I looked in the source code for the SDK, and it seems that only transient ServiceBusException give retries, but I find that quite odd.

UPDATE:

After discussing with my coworkers and looking a little bit more in Application Insights, I can see that it actually has retried 10 times, but all within 2 seconds. This is the Max Delivery Count set on the subscription itself - not on the client. This delivery is not affected by any exponential backoff as I want and need.

score 1 · Accepted Answer · answered Oct 10 '19 at 16:35

I looked in the source code for the SDK, and it seems that only transient ServiceBusException give retries, but I find that quite odd.

This is correct and follows the design. When an error is a transient error (connectivity issue, throttling, etc.) the client will retry using the RetryPolicy. Otherwise, it's not something that a retry would help, therefore re-trying the same operation would not help.

To your specific case - the code you have is executed and the message is processed. There's no issue between the client and the broker. That's why the policy is not kicking in. Further on that, you confirm that it's an application issue, given that the message is retried by your process and ends up in the dead-letter queue.

I am struggling with messages ending up in dead letter queue too quickly.

What you need is an application retries and backoff to ensure the message is not retried immediately multiple times, causing it to be dead-lettered. This part can be a bit tricker, depending on how long your SQL server is down. There are a few options:

Schedule the message in the future to let SQL Server recover. This option will require to schedule a new message.
Defer the message and handle it later. This will mean you'd need to keep a record of the deferred message SequenceNumbers. One way to implement this option is by scheduling a new message with the sequence number of the original message.
Use an abstraction on top of Service Bus that provides a high-level concept/feature to perform retries, e.g. MassTransit or NServiceBus (Recoverability).

Will it be possible to wrap my own (transient) exceptions in ServiceBusExceptions, setting IsTransient to true? Maybe more proper handling is to autoforward from the subscription to a queue which I listen to instead. When application specific transient exception occur, put the same message back on the queue, scheduled for the future, and use a custom message property to handle number of retries and backoff times. — Alf Kåre Lefdal, Oct 11 '19 at 12:08
No, it's not possible. ServiceBusExceptions are thrown by the broker, not custom code. How you'll handle it is up to you. As long as you don't exceed `MaxDeliveryCount`, the message won't be dead-lettered. — Sean Feldman, Oct 11 '19 at 17:50

What kind of exceptions give retries in Azure Service Bus SubscriptionClient?

1 Answers1