1

I'm simply trying to work out how best to retrieve messages as quickly as possible from an Azure Service Bus Queue.

I was shocked that there wasn't some way to properly subscribe to the queue for notifications and that I'm going to have to poll. (unless I'm wrong in which case the documentation is terrible).

I got long polling working, but checking a single message every 60 seconds looks like it'll cost around £900 per month (again, unless I've misunderstood that). And if I add a redundant/second service to poll it'll double.

So I'm wondering what the best/most cost efficient way of doing it is.

Essentially I just want to take a message from the queue, perform an API lookup on some internally held data (perhaps using hybrid services?) and then perhaps post a message back to a different queue with some additional information .

I looked at worker roles(?) -- is that something that could do it?

I should mention that I've been looking at doing this with node.js.

David Makogon
  • 69,407
  • 21
  • 141
  • 189
Darren
  • 1,071
  • 1
  • 15
  • 39
  • Worker roles have nothing to do with service bus - they're simply stateless Windows Server VM's with specific ways to configure and scale. – David Makogon Jun 25 '16 at 01:13
  • How did you get your price ? Service bus queue is available for free. and the cost of a message is nothing – Thomas Jun 25 '16 at 06:22
  • Thanks for the responses. I'd seen somewhere I think that worker roles could get notified of queue additions which is what made me ask. For pricing it was under brokered connections for http calls which I'm assuming is what polling is using. Only the first 1000 are free under Standard Tier. https://azure.microsoft.com/en-gb/pricing/details/service-bus/ Have I got that wrong as that would be great. Thanks again. – Darren Jun 25 '16 at 08:03
  • Seems I may have it wrong.. The FAQ suggests that charges are for "the peak number of concurrent brokered connections" and that "Peaks are measured on an hourly basis, prorated by dividing by 744 hours in a month". They give an example as follows: "10,000 devices receive messages from a Service Bus Queue via HTTP, specifying a non-zero time-out. If all devices connect for 12 hours every day, you will see the following connection charges 10,000 HTTP receive connections * 12 hours per day * 31 days / 744 hours = 5,000 brokered connections." – Darren Jun 25 '16 at 10:44
  • So if my understanding is correct, to have a script performing a long poll every 55 seconds (max allowed), that only counts as one "concurrent" connection right? Even if it's polling every 55 seconds? So that's 1 device * 24 hours * 31 days / 744 hours = 1 out of my allowed 1000 right? I suppose the question still stands -- is this the best/only way to do it? – Darren Jun 25 '16 at 10:52
  • you should have a look at webjobs. There is a SDK to receive message from a queue, Especially you can have a look at Azure function that has a nodejs integration https://azure.microsoft.com/en-us/documentation/articles/functions-overview/ – Thomas Jun 26 '16 at 12:42
  • @Darren, concerning the pricing, you should send a request to Microsoft directly so that they can give you exactly the price you will pay for your need. – Thomas Jun 26 '16 at 12:53

2 Answers2

2

Check out these videos from Scott Hanselman and Mark Simms on Azure Queues. It's C# but you get the idea.

https://channel9.msdn.com/Search?term=azure%20queues%20simms#ch9Search

Touches on:

  • Storage Queues vs. Service Bus Queues
  • Grabbing messages in bulk vs. one by one (chunky vs. chatty)
  • Dealing with poison messages (bad actors)
  • Misc implementation details
  • Much more stuff i can't remember now

As for your compute, you can either do a VM, a Worker Role (Cloud Services), App Service Webjobs, or Azure Functions.

The Webjobs SDK and Azure Functions bot have a way to subscribe to Queue events (notify on message).

(Listed from IaaS to PaaS to FaaS - Azure Functions - if such a thing exists).

Azure Functions already has sample code provided as templates to do all that with Node. Just make a new Function and follow the wizard.

If you need to touch data on-prem you either need to look at integrating with a VNET that has site-to-site connectivity back to your prem, or Hybrid Connections (App Service only!). Azure Functions can't do that yet, but every other compute is a go.

https://azure.microsoft.com/en-us/documentation/articles/web-sites-hybrid-connection-get-started/ (That tutorial is Windows only but you can pull data from any OS. The Hybrid Connection Manager has to live on a Windows box, but then it acts as a reverse proxy to any host on your network).

evilSnobu
  • 24,582
  • 8
  • 41
  • 71
  • Thanks for the response. You've answered my question about functions too as I'd seen they could access messages on a queue, but couldn't access on prem data which is a shame. Thanks again. – Darren Jun 26 '16 at 13:42
2

To deal with Azure ServiceBus Queue easily, the best option seems to be Azure Webjob.

There is a ServiceBusTrigger that allows you to get messages from an Azure ServiceBus queue.

For node.js integration, you should have a look at Azure Function. It is built on top of the webjob SDK and have node.js integration :

In the second article, there is an example on how get messages from a queue using Azure Function and nodejs :

module.exports = function(context, myQueueItem) {
    context.log('Node.js ServiceBus queue trigger function processed message', myQueueItem);
    context.done();
};
Thomas
  • 24,234
  • 6
  • 81
  • 125
  • Thanks for your answer. I'd looked at functions but the lack of hybrid connection ruled it out unfortunately. – Darren Jun 26 '16 at 13:43