1

My webapi must perform a set of heavy operations in order to fullfil a request. To minimize processing time I am offload the "view counter increment" to a webjob.

The way I am doing it currently is by enqueueing a message with userId and productId to azure queue storage at the end of each request. The webjob function triggers on new queue messages and after parsing the message it adds values (increments or adds new) to a static concurrent dictionary.

I am not incrementing and writing to azure table because I'd like to use another timer-based webjob to persist values from concurrent dictionary to the table to avoid excessive writes.

While no messages get lost and all values get recorded/added properly into dict, I am seeing very poor performance. It takes over 5 minutes to process 1000 messages. Messages come in bursts and are processed slower in parallel than when I set the BatchSize to 1. My webjob settings are:

config.Queues.BatchSize = 32;
config.Queues.NewBatchThreshold = 100;
config.Queues.MaxPollingInterval = TimeSpan.FromSeconds(2);
config.Queues.MaxDequeueCount = 3;

The performance doesn't seem to suffer due to dictionary collisions as those seem to be processing fast. My timer shows that it takes between 0 and 9ms to process each increment, so 1000 messages should take 9 seconds at most. Is the webjob lagging/idling and is it by design? At what maximum rate can a webjob process the queue? Are there any other settings I can adjust to makes the webjob run faster? Are there other ways to go about this "view increment" task, considering that I can have many messages with different userIds/productIds?

Alex E
  • 735
  • 2
  • 7
  • 15
  • ``Messages come in bursts and are processed slower in parallel than when I set the BatchSize to 1.`` do you mean that it is faster if setting BatchSize to 1? how many minutes does it take to processing these 1000 messages with BatchSize =1? – Fei Han Feb 17 '17 at 09:58
  • Hi Fred, thanks for reply. I just re-ran my test for 1k messages with BatchSize=1 and see that it takes 6 minutes to process, so in parallel is indeed faster. I crossed-out my original statement from the question. Do you think it is still possible to process messages faster than the current rate of 1k/5min in parallel? – Alex E Feb 17 '17 at 17:17
  • Have you enabled queue logging to see what you can glean from there? Instructions at https://learn.microsoft.com/en-us/rest/api/storageservices/fileservices/enabling-storage-logging-and-accessing-log-data. Also, is the process that is enqueueing messages located in the same data center? – Rob Reagan Feb 20 '17 at 00:15
  • Rob, I am running locally to see if it works fast enough. In the Storage Explorer I can see my messages populated in the queue storage and I monitor the web job through the default console window. I am printing the result of each message process into the console to see what is happening. So far, my local results are too slow to consider deployment to Azure. It seems that there's a delay in webjob's process after a set of messages is processed and before the next set is acquired. – Alex E Feb 20 '17 at 13:57

1 Answers1

0

It seems that there's a delay in webjob's process after a set of messages is processed and before the next set is acquired

As far as I know, WebJobs poll the queue with specific polling algorithm. When a message is found, the SDK waits two seconds and then checks for another message; when no message is found it waits about four seconds before trying again. For detailed information, please check Polling algorithm topic in this article.

Besides, you could try to run WebJobs in a Azure App Service web app and scale WebJobs out to multiple instances.

Fei Han
  • 26,415
  • 1
  • 30
  • 41