2

We have a service bus(Standard Plan) queue with 200K+ messages. We wanted to read the message from the service bus in a batch of 2k-5K and insert it into the SQL Db in batches using Azure functions(Every 30 minutes).

Most of the time the ReceiveMessagesAsync returns a few 100s of messages at times it goes to a single digit. I do understand that the maxMessages parameter doesn't guarantee the massage count but still, I would like to know if there is any way to optimize that it returns at least 50% of the max messages count.

Right now, I use the ReceiveMessagesAsync in the loop and hit the Db once I have 5K+ rows to reduce the number of Db calls. Please recommend if there is any other option to optimize this process.

var receivedMessages = await serviceBusReceiver.ReceiveMessagesAsync(maxMessages: 5000, maxWaitTime: TimeSpan.FromSeconds(1));

Prefetch found is set to 5k. I increased the number but that didn't help.

  • Package used: Azure.Messaging.ServiceBus
  • Version: 7.4.0 .Net
  • Framework: .Net 6
HobbyLobbyVS
  • 61
  • 1
  • 10

1 Answers1

2

The short version is no, there is not currently a way to guarantee the minimum number of messages returned.

Additional Context
When messages are requested, the prioritization is to return data to your application quickly so that the locks for the messages that have been received do not expire waiting for the additional messages needed to fill the batch, and so that your application doesn't sit idle.

The client makes the request for the desired batch size to the network transport, which attempts to build a full batch within a ~20 millisecond window from prefetch and the network stream before returning whatever messages are currently available. If no messages are available within the maxWaitTime that you've specified, nothing is returned.

Prefetch can help by making more messages available to the batch, but they have to be streamed in over the network. Depending on the size of messages, it can take a bit to stream in enough to fill the prefetch cache and if you're consuming messages faster than the network can stream them, the cache will run dry.

An important consideration for prefetch is to remember that messages held in the prefetch cache are locked by the service, and those locks will expire if they are not consumed quickly enough.

Thoughts and Next Steps
In your scenario, it seems as if the application may be able to consume and process messages more quickly than the network can stream them to keep the cache full.

If minimizing database calls is your goal, setting prefetch to your batch size and collecting messages from several receive calls into a batch of 5,000 may be the simplest and safest approach to doing so. Depending on how quickly you meet your batch size, you may need to renew the locks of the messages that you're holding.

Another possible option, though I wouldn't go this way, would be to consider increasing prefetch and introducing a delay between receive calls to let the cache refill; the challenge here is that you lose visibility into what message locks you're responsible for and would be unable to renew locks, so it would not be as reliable.

Jesse Squire
  • 6,107
  • 1
  • 27
  • 30