How to mock ServiceBusClient with retry options

Question

I'm trying to ensure that the retry functionality is working correctly for ServiceBusSender.SendMessageAsync(). After a number of retries I need to do something else.

Currently I'm doing the following to mock both the ServiceBusClient and ServiceBusSender but when I step through the code I can't see SendMessageAsync() being called the number of retries I would expect.

var serviceBusClientMock = new Mock<ServiceBusClient>(
    It.IsAny<string>(), new ServiceBusClientOptions()
    {
        RetryOptions = new ServiceBusRetryOptions()
        {
            Mode = ServiceBusRetryMode.Fixed,
            Delay = TimeSpan.FromSeconds(3),
            MaxDelay = TimeSpan.FromSeconds(10),
            MaxRetries = 3
        }
    }
);

var serviceBusSenderMock = new Mock<ServiceBusSender>();
serviceBusSenderMock
    .Setup(x => x.SendMessageAsync(It.IsAny<ServiceBusMessage>(), It.IsAny<CancellationToken>()))
    .Throws(new ServiceBusException("Region Down", ServiceBusFailureReason.ServiceTimeout));   

serviceBusClientMock
    .Setup(x => x.CreateSender(It.IsAny<string>()))
    .Returns(serviceBusSenderMock.Object);

Why would you test Microsoft library ? you should focus on testing your own code I think. — Thomas, Aug 04 '22 at 08:24
I don't want to test the Microsoft library. I have code that happens once the MaxRetries value has been reached. So I need to mock the client and sender so they have the right responses so that code pathway can run. — AJames, Aug 04 '22 at 08:25
Could you share part of your function that you d like to test ? — Thomas, Aug 04 '22 at 08:27
Part of the other reason why I want to mock the ServiceBusClient is to understand what SendMessageAsync will return once the max number of retries has been reach. — AJames, Aug 04 '22 at 08:35
but if it s a mock, you wont have the real result from SendMessageAsync . — Thomas, Aug 04 '22 at 08:45
True...my bad. Is there another way to simulate a topic being unavailable to see what SendMessageAsync will return after the max retries are reached? Or documentation somewhere? — AJames, Aug 04 '22 at 08:49
not sure what would happen if you provide a wrong namespace ? Azure chaos studio could be an option but not sure if you can simulate servicebus failure tho — Thomas, Aug 04 '22 at 08:52
Providing the wrong namespace gives a different exception entirely unfortunately — AJames, Aug 04 '22 at 08:53
you could check this link: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/servicebus/Azure.Messaging.ServiceBus#troubleshooting and this one as well: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-exceptions — Thomas, Aug 04 '22 at 08:54
Thank you. I found this "IsTransient : This identifies whether or not the exception is considered recoverable. In the case where it was deemed transient, the appropriate retry policy has already been applied and retries were unsuccessful." This sounds like the retires have already occurred within the SendMessageAsync() method so I won't be able to see them occurring in my code. — AJames, Aug 04 '22 at 08:59
This is very tricky as the client is not designed for this scenario. — Sean Feldman, Aug 04 '22 at 11:50

Jesse Squire · Accepted Answer · 2022-08-05T12:53:09.227

Much of this has been teased out via comments, but to offer confirmation:

This sounds like the retires have already occurred within the SendMessageAsync() method so I won't be able to see them occurring in my code.

This is correct.

Client retries are implicit; your application has no idea that they're taking place other than the operation takes longer to complete. In this case, when control returns to your application after the "SendMessageAsync" call, all retries have already been applied. If your application sees an exception, either that exception was terminal (and not retried) or all of the retry attempts were exhausted before the operation was successful.

The easiest way for your application to observe exceptions that are not otherwise surfaced is through the SDK logs. Context for enabling them can be found in this sample.

You could also create a custom retry policy that you'd pass via the ServiceBusClientOptions used to create your client. The RetryOptions member there has a CustomRetryPolicy property that allows you to specify your own implementation of the abstract ServiceBusRetryPolicy class. This would allow your application to observe all exceptions and own the decision for whether or not to retry. That said, unless you copied logic from our built-in BasicRetryPolicy, you would potentially be altering standard behavior.

not sure what would happen if you provide a wrong namespace

As mentioned in the comments, the best place to understand error scenarios is by taking a look at the Exception Handling section of the Overview docs.

If you've got the wrong namespace configured or the client cannot reach the namespace, you'll either see an exception from the .NET networking stack, a TimeoutException or an OperationCanceledException depending on where in the network/transport stack the exception originates from.

In the case of an incorrect a topic, you would see a ServiceBusException with its Reason property set to ServiceBusFailureReason.MessagingEntityNotFound. This is not retriable.

Simulate a topic being unavailable

This isn't a documented scenario by the service, as entities such as topics are monitored for health and will automatically recover. If a topic is unavailable for more than a couple of seconds, it is very likely that there is something very wrong and impacting multiple data centers.

During any migration or recovery of a topic node, the service call will experience a transient failure (either timeout, service busy, or a general communication failure) - which will be handled by the client retry policy.

One caveat to mention here is that it is possible to manually disable a topic via the Azure Portal. In this case, the failure reason for the ServiceBusException is set to MessagingEntityDisabled. This is not retried and it requires manual action to resolve.

Following-up, it looks like the most common manifestation of a namespace that doesn't exist is in the form of a `System.Net.Sockets.SocketException` with the message "No such host is known." — Jesse Squire, Aug 05 '22 at 12:52

How to mock ServiceBusClient with retry options

1 Answers1