In an attempt to catch and retry transient exceptions when sending to a queue in an azure service bus, I am seeing more messages than sent when sending more than 25,000 messages in a test scenario. I expected that if I did have excess messages, they would be the result of some sort of timeout/retry that actually had reached the message bus the first time.
We are using the .Net Standard Micrososft.Azure.ServiceBus library version 3.0.2
The basic logic in an infinite loop:
- await sending message to queue
- break if successful
- catch exception and test if it is a transient one that benefits from retry
- wait random time from 1-3 seconds and retry
- any not caught exception is logged by the caller.
Other notes - in debug, I see many "system.timeout" exceptions, but can't catch them when not running in debug.
public async Task SendToAzureQueue(Message messageToSend) {
while (true) {
try {
await messageSender.SendAsync(messageToSend);
break;
}
catch (Exception ex) when (
ex is ServerBusyException ||
ex is ServiceBusCommunicationException ||
ex is ServiceBusTimeoutException ||
ex is SessionCannotBeLockedException ||
ex is SocketException
) {
_logger.Warn($"Transient Exception Thrown. MessageId: {messageToSend.MessageId} - Retrying in 1-10 seconds.\n", ex);
var randomGenerator = new Random();
await Task.Delay(randomGenerator.Next(1000, 10000));
}
}
}
The calling function wraps this with a try/catch (exception)
Here is the code of the integration test that calls the above code:
var graphLogObjects = AutoFixture.CreateMany<GraphlogWithNumericId>(30000);
var tasks = graphLogObjects
.Select(graphLogEntryDto => _jsonSerializer.Serialize(graphLogEntryDto))
.Select(dataToSend => graphLogMessageQueue.SendToAzureQueue(Message messageToSend));
await Task.WhenAll(tasks);