Azure Service Bus, determine if OnMessage stops processing

Question

Using Azure ServiceBus and the OnMessage call I am looking for a way to determine if the OnMessage event pump stops reading from the queue.

Our connection to OnMessage is configured as below:

protected virtual void DoSubscription(string queueName, Func<QueueRequest, bool> callback)
{
    var client = GetClient(queueName, PollingTimeout);
    var transformCallback = new Action<BrokeredMessage>((message) =>
    {
        try
        {
            var request = message.ToQueueRequest();
            if (callback(request))
            {
                message.Complete();
            }
            else
            {
                message.Abandon();
                Log.Warn("DoSubscription: Message Failed to Process Gracefully: {0}{1}", Environment.NewLine, JsonConvert.SerializeObject(request));
            }
        }
        catch (Exception ex)
        {
            Log.Error("DoSubscription: Message Failed to Process With Exception:", ex);
            message.Abandon();
        }
    });
    var options = new OnMessageOptions
    {
        MaxConcurrentCalls = _config.GetInt("MaxThreadsPerQueue"),
        AutoComplete = false,
        AutoRenewTimeout = new TimeSpan(0,0,1)
    };
    options.ExceptionReceived += OnMessageError;
    client.OnMessage(transformCallback, options);
}

The problem we are encountering is after a period of time with no messages being queued new messages that are queued fail to be picked up by the OnMessage event pump until the application is restarted.

I realize there are ways to do this using Worker Roles, however for monitoring and management purposes we decided to implement this in the Application Start of a web app.

Did you check deadletter queue? If there are messages there, you should be able to see the reasons for your message processing to fail. — mert, Jul 13 '15 at 08:00
In the DL they just timeout. Programatically the process just stopped. — VulgarBinary, Jul 30 '15 at 18:23

score 7 · Accepted Answer · answered Jul 16 '15 at 21:01

So after a call with Microsoft's Azure support team there is not an event to trap when OnMessage or OnMessageAsync errors. As these are not blocking calls, it starts the event pump and returns to the executing thread, this creates a challenge to determine if OnMessage* is doing it's job.

Suggestions from Microsoft were:

Create your own implementation of the QueueClientBase class which exposes the OnClose, On* methods which you could handle. However, in doing this you have to do the handling of the message envelope yourself.
Use OnReceive in a separate thread loop, which you can trap errors yourself and immediately retry.

However, I did explore some bullet proofing against OnMessage and discovered a few things which have eased my fears.

OnMessage is incredibly fault tolerant
- I uplugged the ethernet cable from my laptop with wireless turned off, this broke the OnMessage's connection to the Service Bus queue. After waiting 10 minutes I plugged the ethernet cable back in and the OnMessage immediately began processing queued elements.
On Message surprisingly is fairly stable. It has been running inside the global.asax.cs App Start, to abbreviate, for days on end with a Factory IdleTimeout set to 24 hours without restarting the web application for 72 hours.

All-in-all I'm going to continue using OnMessage/OnMessageAsync for now and keep an eye on it. I will update this if I see issues that change my opinion of OnMessage.

Aside - Make sure if you are using OnMessage for permanent listening in an Azure Web Site that you set the "Always On" configuration option to "On". Otherwise, unless a web request comes in OnMessage will be disposed and messages will no longer be processed until the web application is reawakened by a HTTP request.

After over a month of uninterrupted processing `OnMessage` has been working without issue. At this junction I would say that it is acceptable to use OnMessage without the capability of determining stopped state without great concern. To be safe we have a redundancy in place to check the oldest element in the queue, if it's age is over a configured age it will notify us. The findings plus a watch for notification is sufficient in our case to continue with this approach. — VulgarBinary, Aug 17 '15 at 20:55
Another follow up... it's now been over a year. The solution with OnMessage has still not faltered or failed. I stand by my initial statement, OnMessage is resilient enough for unaltered production use. — VulgarBinary, Oct 19 '16 at 19:33
in what process do you instantiate the QueueClient ? And how do you prevent garbage collection? — Matt Evans, Oct 27 '17 at 08:06
When you say you set the factory IdleTimeout to 24 hours; where did you set this? — Jono Rogers, Dec 31 '17 at 11:58

Azure Service Bus, determine if OnMessage stops processing

1 Answers1

Linked