Robust Retry Handling for Azure Message Queues & Multi Instance Worker Roles

Question

Reposted from CodeReview.Stackexchange as it's only pseudo code

I'm building a task processor worker role. Tasks are inserted into a simple DB Table (Guid, Json string, Status) and then the Guid is pushed onto an azure message queue. The worker role will then pop the message from queue, retrieve the JSON instructions from the DB & process that task.

I'm trying to ensure I've covered off all the corner cases for idempotency & retries and have come up with the following pseudo code.

My main issue is that my "ProcessTask" phase is non-transactional (HTTP Posts & Service Calls to external services) where repeated calls would be Bad (Mkay)

Can someone double check this and see if there are any corner cases I haven't identified.

Lets assume a task exists in the database with the following info.

| Id   | TaskInstruction       | Status  | PopReceipt | 
+------+-----------------------+---------+------------+
| GUID | { ... some json ... } | Pending | NULL       |

The following pseudo code covers the following situations.

A worker retrieves a message, processes it in time, updates the db and deletes the message.
#1 except it takes longer than the 30 second invisibility timeout but still succeeds.
A message fails to be processed and needs to be retried.
A message fails to be processed many times and exceeds the max allowed dequeue count.

The plan is to make use of the Azure BlobStorage AquireLease mechanism to use as a File Lock around critical pieces of code for idempotency reasons.

PSEUDO-CODE

BlobAquireLease                                  //LOCK Only Worker Role Can Retrieve at a time
    var message = Queue.GetMessage(30);          // 30 second invisibility timeout
    var dbTask = DBSession.GetTask(message.Id);  //Retrieve the DB Entry
    If dbTask.Status != Pending OR Retrying
        //Crap another role is processing this task but has 
        //overshot the 30 second timeout and now my popreceipt 
        //supercedes his - put my popreceit in the DB, and do nothing more
        DBSession.UpdateTask(message.Id, newPopReceipt) // put the new pop receipt in the db
        BlobReleaseLease                                //Release the LOCK
        return;                                         //Get out of here
    EndIf

    //Otherwise, I'm ok to handle this task.
    DBSession.UpdateTask(message.Id, Status.Processing);
BlobReleaseLock

//At this point, the task is mine, and I've got 30 seconds to do stuff

TRY
    ProcessTask(dbtask);
    //Successful
    BlobAquireLease                                                //Grab the LOCK Again
        //Get from DB Again just incase I've 
        //over shot the 30 seconds and someone else has updated the popreceipt
        var dbTaskRefreshed = DBSession.GetTask(message.Id); 
        DBSession.UpdateTask(message.Id, Status.SUCCESS);          //Update the status in the DB
        Queue.DeleteMessage(message, dbTaskRefreshed.PopReceipt);  //Delete the message
    BlobReleaseLease
ENDTRY
CATCH
    //An Error Occured and the task couldn't be processed. 
    //need to update the database
    BlobAcquireLease                                               //Grab the LOCK again

        IF message.DequeueCount > MAX ALLOWED                      //Too many Attempts
            //Get from DB Again just incase I've 
            //over shot the 30 seconds and someone else has updated the popreceipt
            var dbTaskRefreshed = DBSession.GetTask(message.Id); 
            DBSession.UpdateTask(message.Id, Status.FAILED);
            Queue.DeleteMessage(message, dbTaskRefreshed.PopReceipt)
        ELSE
            //Haven't reached the max dequeue count yet, so just 
            //let the invisibility timeout elapse on it's own and someone will retry
            DBSession.UpdateTask(message.Id, Status.RETRYING);
        END IF
    BlobReleaseLease
ENDCATCH

Robust Retry Handling for Azure Message Queues & Multi Instance Worker Roles

0 Answers0