4

What is the best way to scale a Worker Role that is processing many long running Azure Service Bus messages using the QueueClient Message Pump.

If using QueueClient.OnMessageOptions.MaxConcurrentCalls = 6 and QueueClient.OnMessage does that mean i can only process a max of 6 messages at a time?

Is it bad form to have the long running processing within the OnMessage callback to spawn a new Task to complete it's processing?

Should i be using the QueueClient.OnMessageAsync instead?

Thanks for any help.

Lukie
  • 905
  • 1
  • 11
  • 21

1 Answers1

3

By “long running” do you mean IO-bound or CPU-bound?

Assuming IO-bound then I wouldn’t spawn a new Task in the OnMessage callback. This creates thread management overhead that can slow processing down at scale.

Consider using OnMessageAsync if you are using IO-bound operations and make sure that you await the asynchronous implementations of any of these operations. This uses your existing threads much more efficiently.

If your operations are CPU-bound then Task creation may do more for you. The mechanics of this are discussed in a series of excellent posts by Stephen Cleary:

http://blog.stephencleary.com/2013/10/taskrun-etiquette-and-proper-usage.html

The MaxConcurrentCalls property controls the number of concurrent requests to the service bus. Increasing this number has a limited impact if you’re IO-bound and limited by available bandwidth. I would recommend doing a bit of performance testing with the Azure client-side performance counters to get the optimum value for your environment.

Ben Morris
  • 585
  • 3
  • 6
  • 1
    AFAIK `MaxConcurrentCalls` setting only affects how many concurrent callback threads the `QueueClient` will issue, and that there is a single request made to ASB no matter that number. ie: `OnMessage` is not a polling-driven interface in so much as it is not simply wrapping `QueueClient.Receive` in multiple threads. It's more like `ReceiveBatch -> foreach received, spawn threads while active count < Max` – JoeBrockhaus Jun 03 '15 at 23:24