2

Our application dequeues the message from Azure queue and makes it invisible for a given period of time.

Worker than processes it and when all work is done it deletes it from the queue.

But sometimes delete fails with 404 error not found. The problem might be that message's pop receipt has been changed.

Because when message is dequeued separate thread also runs and increases invisibility of a message to prevent it to be picked by other consumer. But it calls UpdateMessage which actually changes pop receipt.

Because UpdateMesasge and DeleteMessage might run at the same time, DeleteMessage sometimes fails because its PopReceipt is no longer valid.

Is there any way to avoid PopReceipt change on UpdateMessage?

Code sample:

TimerCallback extenderHandler = new TimerCallback(async state =>
{
  try
  {
    var oldNextVisibleTime = queueMessage.NextVisibleTime.Value;

    // extend message lease to prevent it from being picked by another worker instances
    await returnMessage();
   }
   catch (Exception ex)
   {
     // NOTE: exceptions on Timer are not propagated to main thread; the error is only logged, because operation will be retried;
   }
});

// start message extender timer which extends message lease time when it's nearing its max hold time, timer runs until it's disposed
using (var messageExtenderTimer = new System.Threading.Timer(extenderHandler, null, 0, (int)MessageLeaseCheckInterval.TotalMilliseconds))
{
  processMessage();
}

In returnMessage method UpdateMessageAsync from Microsoft.WindowsAzure.Storage.Queue is called.

In processMessage method processing itself is done and at the end message is deleted using DeleteMessage method from Microsoft.WindowsAzure.Storage.Queue

And sometimes fails UpdateMessageAsync and sometimes DeleteMessage. Because of that I wonder that when these two concurrent threads make changes to the message - message is changed in the queue before PopReceipt is updated on message itself.

abatishchev
  • 98,240
  • 88
  • 296
  • 433
michal.jakubeczy
  • 8,221
  • 1
  • 59
  • 63
  • `Because when message is dequeued separate thread also runs and increases invisibility of a message to prevent it to be picked by other consumer. But it calls UpdateMessage which actually changes pop receipt.` Does this separate thread in the same worker process of another worker process? – Gaurav Mantri Sep 08 '17 at 09:15
  • @GauravMantri: same worker process and the same CloudQueueMessage instance. – michal.jakubeczy Sep 08 '17 at 09:20

1 Answers1

3

Is there any way to avoid PopReceipt change on UpdateMessage?

Unfortunately no. Whenever a message is updated, a new PopReceipt will be returned. From the documentation link (#4 item):

The message has been updated with a new visibility timeout. When the message is updated, a new pop receipt will be returned.

Gaurav Mantri
  • 128,066
  • 12
  • 206
  • 241
  • Thanks, but this new PopReceipt should be update on the CloudQueueMessage instance i am updating, right? – michal.jakubeczy Sep 08 '17 at 12:33
  • Can you share your code? Without looking at the code, it won't be possible to say that. – Gaurav Mantri Sep 08 '17 at 12:42
  • We are using UpdateMessageAsync from Microsoft.WindowsAzure.Storage.Queue.CloudQueue class. But the problem is that this method does not return anything. So I do not know where do get popreceipt from – michal.jakubeczy Sep 28 '17 at 14:18
  • I have not tried the code but take a look at the properties of CloudQueueMessage that you pass as input. My guess is that this object gets modified with the latest pop receipt. HTH. – Gaurav Mantri Sep 28 '17 at 14:23
  • 1
    you're right, it gets ... so the problem is (I suppose) that there are two parallel threads which work with this message. One which only extends its visibility (thread A) and second one which processes it and deletes it (thread B) from queue at the end. And from time to time this delete on thread B fails. I suppose it happens when thread A is 'extending' its visibility - and thread B wants is removing it. PopReceipt gets changed during these two requests. hoping I explained it clearly. anyway i am not sure whether there is a way to get around this. – michal.jakubeczy Sep 28 '17 at 14:39
  • I suppose the only thing I can do is to make own retry policy which takes 404 as transient error and retries because I think second try will work as PopReceipt gets updated. – michal.jakubeczy Sep 28 '17 at 15:08
  • Without looking at your code, it is hard to make recommendation. But one thing that confuses me is why do you have 2 threads working on the same message. The thread that dequeues it should be the one to process it and subsequently delete it. Also I would not recommend making 404 as a transient error as you can get 404 in a number of cases e.g. message is actually deleted, queue is deleted etc. – Gaurav Mantri Sep 28 '17 at 15:21
  • one thread works on the message processing and second one is just responsible for extending its invisibility in the queue to avoid another threads taking it. i cannot do this in a single thread because operations in single thread might take undefined amount of time and I can miss to extend its invisibility on time. if you have some approach how to extend invisibility in a single thread feel free to share. i know i might make it invisible for a long time when dequeueing but if workers stop it should become visible in a short period of time. – michal.jakubeczy Sep 28 '17 at 18:11
  • I added some code, hoping it sheds some light into it. – michal.jakubeczy Oct 06 '17 at 15:04