How to avoid receiving messages multiple times from a ServcieBus Queue when using the WebJobs SDK

Question

I have got a WebJob with the following ServiceBus handler using the WebJobs SDK:

[Singleton("{MessageId}")]
public static async Task HandleMessagesAsync([ServiceBusTrigger("%QueueName%")] BrokeredMessage message, [ServiceBus("%QueueName%")]ICollector<BrokeredMessage> queue, TextWriter logger)
{
    using (var scope = Program.Container.BeginLifetimeScope())
    {
        var handler = scope.Resolve<MessageHandlers>();
        logger.WriteLine(AsInvariant($"Handling message with label {message.Label}"));

        // To avoid coupling Microsoft.Azure.WebJobs the return type is IEnumerable<T>
        var outputMessages = await handler.OnMessageAsync(message).ConfigureAwait(false);

        foreach (var outputMessage in outputMessages)
        {
            queue.Add(outputMessage);
        }
    }
}

If the prerequisites for the handler aren't fulfilled, outputMessages contains a BrokeredMessage with the same MessageId, Label and payload as the one we are currently handling, but it contains a ScheduledEnqueueTimeUtcin the future.

The idea is that we complete the handling of the current message quickly and wait for a retry by scheduling the new message in the future.

Sometimes, especially when there are more messages in the Queue than the SDK peek-locks, I see messages duplicating in the ServiceBus queue. They have the same MessageId, Label and payload, but a different SequenceNumber, EnqueuedTimeUtc and ScheduledEnqueueTimeUtc. They all have a delivery count of 1.

Looking at my handler code, the only way this can happen is if I received the same message multiple times, figure out that I need to wait and create a new message for handling in the future. The handler finishes successfully, so the original message gets completed.

The initial messages are unique. Also I put the SingletonAttribute on the message handler, so that messages for the same MessageId cannot be consumed by different handlers.

Why are multiple handlers triggered with the same message and how can I prevent that from happening?

I am using the Microsoft.Azure.WebJobs version is v2.1.0

The duration of my handlers are at max 17s and in average 1s. The lock duration is 1m. Still my best theory is that something with the message (re)locking doesn't work, so while I'm processing the handler, the lock gets lost, the message goes back to the queue and gets consumed another time. If both handlers would see that the critical resource is still occupied, they would both enqueue a new message.

What is your messagehandler doing ? sorry but the way you're doing thing sounds weird.... — Thomas, May 07 '18 at 09:14
I have a couple of different message handlers working with that pattern. I cannot disclose what exactly they are doing. What exactly do you think sounds wired? — D. Siemer, May 07 '18 at 13:28
You receive a message then you enqueue a new one in the future. Dont really understand why. could you not process messages as they arrive ? — Thomas, May 07 '18 at 21:01
I enqueue a message to be processed in the future, if the prerequisites of the handler aren't fulfilled. Let's say I have two entities in my system that need to be finished before the operation I trigger with the message can start, I need to check the state of these prerequisites and if they aren't finished, I need to wait a little until I can start processing. — D. Siemer, May 08 '18 at 06:02
Ohh ok make sense now :-) Maybe you should only process one message at a time ? — Thomas, May 08 '18 at 09:29
As you can see in the code above, I set a SingletonAttribute to process one message per message ID at a time. The problem I described in the last paragraph will will occur even then. — D. Siemer, May 08 '18 at 13:01

score 0 · Answer 1 · answered May 14 '18 at 14:53

After a little bit of experimenting I figured out the root cause and I found a workaround.

If a ServiceBus message is completed, but the peek lock is not abandoned, it will return to the queue in active state after the lock expires.

The ServiceBus QueueClient, apparently, abandons the lock, once it receives the next message (or batch of messages).

So if the QueueClient used by the WebJobs SDK terminates unexpectedly (e.g. because of the process being ended or the Web App being restarted), all messages that have been locked appear back in the Queue, even if they have been completed.

In my handler I am now completing the message manually and also abandoning the lock like this:

public static async Task ProcessQueueMessageAsync([ServiceBusTrigger("%QueueName%")] BrokeredMessage message, [ServiceBus("%QueueName%")]ICollector<BrokeredMessage> queue, TextWriter logger)
{
    using (var scope = Program.Container.BeginLifetimeScope())
    {
        var handler = scope.Resolve<MessageHandlers>();
        logger.WriteLine(AsInvariant($"Handling message with label {message.Label}"));

        // To avoid coupling Microsoft.Azure.WebJobs the return type is IEnumerable<T>
        var outputMessages = await handler.OnMessageAsync(message).ConfigureAwait(false);

        foreach (var outputMessage in outputMessages)
        {
            queue.Add(outputMessage);
        }

        await message.CompleteAsync().ConfigureAwait(false);
        await message.AbandonAsync().ConfigureAwait(false);
    }
}

That way I don't get the messages back into the Queue in the reboot scenario.

How to avoid receiving messages multiple times from a ServcieBus Queue when using the WebJobs SDK

1 Answers1