7

I have an Azure WebJob project, that I am running locally on my dev machine. It is listening to an Azure Service Bus message queue. Nothing going on like Topics, just the most basic message queue.

It is receiving/processing the same message multiple times, launching twice immediately when the message is received, then intermittently whilst the message is being processed.

Questions:

  • How come I am receiving the same message multiple times instantly? It seems that it's re-fetching before a PeekLock is applied?
  • How come the message is being re-received even though it is still being processed? Can I set the PeekLock duration, or somehow lock the message to only be processed once
  • How can I ensure that each message on the queue is only processed once?
  • I want to be able to process multiple messages at once, just not the same message multiple times, so setting MaxConcurrentCalls to 1 does not seem to be my answer, or am I misunderstanding that property?

I am using an async function, simple injector and a custom JobActivator, so instead of a static void method, my Function signature is:

public async Task ProcessQueueMessage([ServiceBusTrigger("AnyQueue")] MediaEncoderQueueItem message, TextWriter log) {...}

Inside the job, it is moving some files around on a blob service, and calling (and waiting for) a media encoder from media services. So, whilst the web job itself is not doing a lot of processing, it takes quite a long time (15 minutes, for some files).

The app is launching, and when I post a message to the queue, it responds. However, it is receiving the message multiple times as soon as the message is received:

Executing: 'Functions.ProcessQueueMessage' - Reason: 'New ServiceBus message detected on 'MyQueue'.'
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New ServiceBus message detected on 'MyQueue'.'

Additionally, whilst the task is running (and I see output from the Media Service functionality), it will get another "copy" from the queue.

finally after the task has completed, it's still intermittently processing the same message.

AndrewP
  • 1,598
  • 13
  • 24
  • 1
    What's the DeliveryCount and LockDuration specified on the queue? – Sean Feldman Aug 04 '16 at 12:40
  • I need to look into it. Weird thing about the LockDuration is that it's seemingly picking up the message twice almost instantly, as if it's not locking it at all. I wonder if it's actually an issue with the way I have set up the asynchronous handler? – AndrewP Aug 06 '16 at 06:44
  • Sounds a bit off. The broker will never give the same message more than to one competing consumer if LockDuration has not expired. I suspect something else is happening. Any chance you could share a repro on GitHub or BitBucket? – Sean Feldman Aug 06 '16 at 17:50

1 Answers1

9

I suspect what's happening is the following: Maximum DurationLock can be 5 minutes. If processing of the message is done under 5 minutes, message is marked as completed and removed from the broker. Otherwise, message will re-appear if processing takes longer than 5 minutes (we lost the lock on the message) and will be consumed again. You could verify that by looking at the DeliveryCount of your message.

To resolve that, you could renew message lock just before it's about to expire using BrokeredMessage.RenewLockAsync().

Sean Feldman
  • 23,443
  • 7
  • 55
  • 80
  • I will check out that method, and will look into the delivery count and duration lock. Weird thing is that it's not happening every time, and there doesn't appear to be much consistency about when it is happening, and also that it appears to be processing the same message twice immediately. But thanks for pointing out that function! – AndrewP Aug 06 '16 at 06:46
  • @AndrewP It could also be a case of auto-retry if your job is throwing an unhandled exception, and you have autocompletemessage true in the service bus config. You can test this by simply throwing an ex in your trigger function, then post one message to the topic - you should see a message detection up until the max # of retries. – JoeBrockhaus Aug 23 '16 at 22:20
  • Isn't the lock automatically renewed by default? So after 5 minutes, the lock will automatically be renewed if the processing is still ongoing. Or am I missing something? – GETah Nov 05 '16 at 06:47
  • Not automatically. If that would happen, you could lock out your resources by never finishing the processing. C# client has that option, but even that has a limit you need to specify. – Sean Feldman Nov 05 '16 at 14:32