Azure Service Bus exception - MessageLockLostException: The lock supplied is invalid. Either the lock expired, or the message has already been removed

Question

By using the Azure service bus topics and Subscriptions - I am able to pass message between the two systems. But however sometimes, I get this lock expired exception. How to avoid it ?

Exception - Message handler encountered an exception Microsoft.Azure.ServiceBus.MessageLockLostException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue.

Below is the code :

static async Task MainAsync()
        {
           TokenProvider tokenProvider = TokenProvider.CreateManagedIdentityTokenProvider();

            subscriptionClient = new SubscriptionClient(serviceBusNamespace, topicName, subscriptionName, 
            tokenProvider, receiveMode: ReceiveMode.PeekLock);

            // Register subscription message handler and receive messages in a loop
            RegisterOnMessageHandlerAndReceiveMessages();
            Console.Read();

            await subscriptionClient.CloseAsync();
        }


 static void RegisterOnMessageHandlerAndReceiveMessages()
        {
            var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
            {
                MaxConcurrentCalls = 1,
                AutoComplete = false
            };

            // Register the function that processes messages.
            subscriptionClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
        }

static async Task ProcessMessagesAsync(Message message, CancellationToken token)
        {
            // Process the message recieved.


            Console.WriteLine($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
            await subscriptionClient.CompleteAsync(message.SystemProperties.LockToken);

            }

Answer 1

Lock lost exception generally is a client-side error, there are two possible causes, either there was some issue at the service bus end (high utilization, noisy neighbor, etc. ) or the client is too busy that it was unable to complete the request in a timely manner or some very slow network.

Check the service bus metrics from portal and see if there is any internal or server busy error during that time when you have observed the issue. If not then the suggestion would be looking into your client application metrics and see if there is any high utilization during that time (CPU, memory, network etc.) that might cause issues during that time.

If the above doesn't help and still observing the lock lost error more frequently then I suggest you to add logging in your application code to confirm how much time it is taking for the message to get completed. If there is any business logic that you are executing then please log the time how much time it takes to execute that piece of code. The log should give you the details what was the messageID that your application is consuming, timeframe in UTC, lock token that is acquired on that message along with the execution time for each and every statement (business logic) You can also add a timer before and after CompleteAsync call to check how much time it is taking for the message to be marked as completed. I will also suggest you to to remove the prefetch if you have added any in your code.

Verifying the above details if the issue is happening within the lock duration then this would need in-depth troubleshooting from both client and service bus end. Once you have the above information and added more logging in your application you can either create a support ticket for further assistance or please send an email with the subject line “Attn:Mayank” to AzCommunity[at]Microsoft[dot]com referencing this thread along with your Azure subscription ID, service bus name, queue name, receiver application details along with the exception message, message ID, lock token, timeframe for individual messages for which you have observed the issue.