0

I'm developing a system based on Azure Service Bus for quick fire-and-forget through an API and async processing of a lot of messages by background services through a topic. In the context of this question, the topic has a single subscription, why it could have been a queue. For other reasons, I would like to keep this as a topic.

I recently migrated the code from .NET framework app using the WindowsAzure.ServiceBus package to a .NET Core package using the Microsoft.Azure.ServiceBus package. To process a large number of message, I'm using the MessageReceiver class like this:

var connString = "...";
var subscriptionPath = EntityNameHelper.FormatSubscriptionPath("topic", "subscription");
var messageReceiver = new MessageReceiver(connString, subscriptionPath);
while (...)
{
    var messages = await messageReceiver.ReceiveAsync(10, TimeSpan.FromSeconds(5));
    ...
}

For simplicity, I have hidden a range of details. Like the fact that my app starts 5 threads and process messages in each thread using the same messageReceiver instance.

I normally have more than one instance of this app running to spread out both across threads and processes. And I believe we have finally arrived at the code of my question. After migrating to .NET Core and the new NuGet package, I noticed that only one of the apps is processing messages at the same time. When opening two console windows and launching a process in each window, I can see the app in window 1 starts processing. App in windows 2 doesn't process anything. After a number of seconds, the app in windows 1 stops processing and the app in window 2 starts processing. After a while, it switches back. There's no real pattern in the switch, but all of my messages are successfully processed.

Is there some kind of limitation in MessageReceiver that allows a total maximum number of threads process messages from the same subscription or something like that?

ThomasArdal
  • 4,999
  • 4
  • 33
  • 73

1 Answers1

1

I'm not aware of any limitations with MessageReceiver when it comes to the number of threads. The new library though was optimized to take advantage of concurrency w/o the need for threads (asynchronous code). So technically you could run with a single thread and have multiple concurrent receiving tasks. Another approach would be to use the message handler provided by QueueClient and SubscriptionClient that allow specifying concurrency for processing multiple messages easily, but those allow receiving one message per concurrent callback (no batching).

The broker provides as many messages as it can in a single call to the first competing consumer. If there are not enough messages, all the messages will be given to a single (or a first few) consumers. There's no round-robin and fair distribution. It works as expected indeed.

Sean Feldman
  • 23,443
  • 7
  • 55
  • 80
  • Thanks! I did try `SubscriptionClient` but as you write, no batching. Batching is essential in my case since needing to receive and ack each message individually is too expensive in terms of ms when communicating with the API. I'm not sure `MessageReceiver` support multi-thread callbacks like `SubscriptionClient`. Is this something you know is documented somewhere? – ThomasArdal Jan 23 '20 at 17:34
  • I learned a bit more by inspecting the consoles under different load. Now there's a lot of load on the topic. All apps (running 4 now) are consuming messages. When load goes down, two are consuming all the time and 2 are idle. I guess this is working as it should, though different from back when I used the old package. – ThomasArdal Jan 23 '20 at 18:58
  • 1
    Correct. The broker provides as many messages as it can in a single call to the first competing consumer. If there are not enough messages, all the messages will be given to a single (or a first few) consumers. There's no round-robin and fair distribution. It works as expected indeed. – Sean Feldman Jan 23 '20 at 19:57
  • Sounds good. Would it make sense to copy your last comment as part of the answer? I will mark it as the correct answer – ThomasArdal Jan 24 '20 at 06:39
  • Not a bad idea. Done. – Sean Feldman Jan 24 '20 at 17:48
  • @SeanFeldman I have a .net core API which has a receiver and was working properly and all of a sudden the messages are not processed anymore. I have the completeasync and abandon asyc implemented. My current receiver is registered as InstanceperLifetimeScope, I am thinking if changing it to InstancePerLifetimeScope can help. – Darey Sep 25 '20 at 16:13
  • @Darey, hard to know what is it w/o some more context and code. If you can reproduce the issue, might be a bug and I would suggest raising an issue with the client SDK team [here](https://github.com/Azure/azure-sdk-for-net/issues). Otherwise, maybe a separate SO post with more details to help with understanding what's happening and what the setup looks like. – Sean Feldman Sep 27 '20 at 17:29
  • @seanFeldman Thank you sir, I right now caught the exceptions and logging it than throwing it to DLQs. But will keep this under observation. – Darey Sep 29 '20 at 07:20