3

In an attempt to catch and retry transient exceptions when sending to a queue in an azure service bus, I am seeing more messages than sent when sending more than 25,000 messages in a test scenario. I expected that if I did have excess messages, they would be the result of some sort of timeout/retry that actually had reached the message bus the first time.

We are using the .Net Standard Micrososft.Azure.ServiceBus library version 3.0.2

The basic logic in an infinite loop:

  1. await sending message to queue
  2. break if successful
  3. catch exception and test if it is a transient one that benefits from retry
  4. wait random time from 1-3 seconds and retry
  5. any not caught exception is logged by the caller.

Other notes - in debug, I see many "system.timeout" exceptions, but can't catch them when not running in debug.

  public async Task SendToAzureQueue(Message messageToSend) {

        while (true) {
            try {                    
                await messageSender.SendAsync(messageToSend);
                break;
            }
            catch (Exception ex) when (
                ex is ServerBusyException ||
                ex is ServiceBusCommunicationException ||
                ex is ServiceBusTimeoutException ||
                ex is SessionCannotBeLockedException ||
                ex is SocketException
            ) {
                _logger.Warn($"Transient Exception Thrown. MessageId: {messageToSend.MessageId} - Retrying in 1-10 seconds.\n", ex);
                var randomGenerator = new Random();
                await Task.Delay(randomGenerator.Next(1000, 10000));
            }                
        }
    }

The calling function wraps this with a try/catch (exception)

Here is the code of the integration test that calls the above code:

var graphLogObjects = AutoFixture.CreateMany<GraphlogWithNumericId>(30000);

var tasks = graphLogObjects
            .Select(graphLogEntryDto => _jsonSerializer.Serialize(graphLogEntryDto))
            .Select(dataToSend => graphLogMessageQueue.SendToAzureQueue(Message messageToSend));

await Task.WhenAll(tasks);
DanCaveman
  • 676
  • 1
  • 6
  • 22
  • Are send and receive operation concurrent or not? What's the recieve processing logic looks like and how long does it take to complete a message? How do you recieve your messages, using OnMessage API/message handler or receiving manually? If manually, do you receive in batches? Is there a prefetch configured? etc, etc, etc. It would be really helpful to see your code and not just a snippet for sending a single messages. Thanks. – Sean Feldman Aug 08 '18 at 22:32
  • @SeanFeldman - Yes, the sending logic is concurrent. I will post the logic shortly. As for receiving, that is a separate service that is turned off. I am just looking (with queue explorer) at the counts after running a test that creates 30,000 messages and sends them in concurrently. When changed to synchronous, the number is 30,000. I suspect a timeout occurs and is automatically resent even though the queue already got it. – DanCaveman Aug 09 '18 at 16:17
  • @SeanFeldman - it takes a fraction of a second to send the message. I haven't put timers around it, but that definitely contributes to the problem. When I send the same number of integers, the correct number of integers arrive in the queue. – DanCaveman Aug 09 '18 at 16:25
  • do you get any exceptions logged in the SendToAzureQueue() method? Also, what ASB client and version are you using? – Sean Feldman Aug 09 '18 at 17:51
  • @SeanFeldman - MessageSender in "Microsoft.Azure.ServiceBus" version 3.0.2 - There is not exception caught/logged in the SendToAzureQueue() method. Even a "catch all" around the entire method does not log any errors. When I watch the test in Debug - I see a System.Timeout exception. I am not resubmitting those though. If they are getting submitted more than once - could it be some auto retry in the sender? – DanCaveman Aug 09 '18 at 19:11
  • `MessageSender` retries if there's a failure to get an acknowledgement from the broker. I would consider it a bug if there was no acknowledgement, but message stored on the service. Any chance you could share a GitHub project with the repro? I'll run it locally as well. – Sean Feldman Aug 09 '18 at 20:17

0 Answers0