11

I have a queue in Azure storage named for example 'messages'. And every 1 hour some service push to this queue some amount of messages that should update data. But, in some cases I also push to this queue message from another place and I want this message be proceeded immediately and I can not set priority for this message.

What is the best solution for this problem? Can I use two different queues ('messages' and 'messages-priority') or it is a bad approach?

Bogdan Kolodii
  • 299
  • 3
  • 11
  • With Windows Azure Queues, you can't set the priority message so there your only option would be to use 2 separate queues. May I suggest you take a look at Windows Azure Service Bus Queues? I haven't used it personally but I believe it is more feature rich than Windows Azure Queues. – Gaurav Mantri Jan 19 '14 at 16:32
  • I haven't used Windows Azure Service Bus Queues too, but will try to investigate it. – Bogdan Kolodii Jan 20 '14 at 11:57
  • https://msdn.microsoft.com/en-us/library/dn589794.aspx?f=255&MSPPError=-2147217396 – vladimir Feb 11 '16 at 18:57

1 Answers1

11

The correct approach is to use multiple queues - a 'normal priority' and a 'high priority' queue. What we have implemented is multiple queue reader threads in a single worker role - each thread first checks the high priority queue and, if its empty, looks in the normal queue. This way the high priority messages will be processed by the first available thread (pretty much immediately), and the same code runs regardless of where messages come from. It also saves having to have a reader continuously looking in a single queue and having to be backed off because there are seldom messages.

Simon Munro
  • 5,399
  • 6
  • 33
  • 40
  • 2
    Isn't there a starvation issue with this approach? Given enough high priority messages, the normal message queue will never be served? – urig May 20 '14 at 09:02
  • 2
    Good point, but then 'high priority' becomes the new 'normal', and you have an external set of problems that is causing to many high priority messages - that is a major business issue or panic that needs to be addressed. Our system records data for vehicle tracking and 'high priority' is for crash detection (which sends telemetry data to emergency services). In that case there is **no way** that high priority would exceed normal unless there was a bug or the world was coming to an end – Simon Munro May 21 '14 at 11:09
  • 2
    How can you handle scaling in Azure when you have two queues and the auto scale only accounts for one? – Jazaret Oct 12 '15 at 18:00
  • @Jazaret You autoscale on the 'normal' priority. See previous comment. If you have to autoscale high priority, then you have a different problem... in which case you have to queue reader processes than can scale independently. This technique is for the occasional high priority message where you won't want the overhead of a separately managed (and scaled) queue reader – Simon Munro Sep 28 '16 at 15:17