0

I have 2 WebJobs in my WebApp. Let's call them WJ1 and WJ2, both WebJobs are triggered by queues (Q1 and Q2 respectively). Since Q2 gets a lot of messages and I would like to process them quickly, I scale out my WebApp to 10 instances (it only runs with 1 instance at times when no messages are expected).

The problem is that since Q2 receives overwhelmingly more messages than Q1, WJ2 takes all the resources and the processing of Q1 lags behind its expected schedule. Would there be a way of assigning WJ1 a higher priority than WJ2, so that whenever there is a message in Q1 it will be process before taking any more from Q2?

Both WebJobs are separate (and both have 2 functions each, triggered by the afore mentioned queues and timers) and can be started and stopped independently, if that is of any help.

Also, WJ1 would be happy with just one instance, since the messages are expected to be processed one after the other.

I have read and thought about splitting the WebJobs in 2 different WebApps and limit the WebApp2 to 9 of the 10 available instances to run WJ2. However, I don't like that option because WJ2 takes 3 hours or so to finish its burst of messages and WJ1 should be done within an hour or less, so it makes no sense to prevent WJ2 to take all the available capacity once WJ1 is done.

Also, if WJ1 was a singleton, would it then have more prio? (Probably not, but worth asking).

Thanks!

gris
  • 131
  • 1
  • 11

1 Answers1

0

Simply scaling the compute resources might not be sufficient to prevent loss of performance under load. You might also need to scale storage queues and other resources to prevent a single point of the overall processing chain from becoming a bottleneck. Also, consider other limitations, such as the maximum throughput of storage and other services that the application and the background tasks rely on.

Pls go through Scaling and performance considerations of Web Jobs.

Secondly, you should try to keep your webjobs smaller - 3 hours and 1 hour are very huge timespans. It means that your WJ2 might acquire all the resources for 3 hours! It requires in depth performance review of the web job and identify if any/how many resource(s) is/are being held by the webjob unnecessarily. Smaller WebJobs will allow different instances to scale independently as per their own usage and free resources asap.

Harshita Singh
  • 4,590
  • 1
  • 10
  • 13
  • the webjobs are not running for 1 or 3 hours. Consuming all the messages in their respective queues takes 1 hour for WJ1 and 3 hours for WJ2 when they are not competing with one another for resources. The thing is that the messages come relatively at the same time of the day for both WJs – gris Nov 26 '20 at 08:00
  • Then, the better thing is to convert your WJ1 to a separate Azure function as it is billed on need basis only. And keep a separate App service for WJ2. – Harshita Singh Dec 03 '20 at 05:12
  • Does this help you @gris? – Harshita Singh Dec 10 '20 at 11:25
  • hi, thanks for the reply but we considered and discarded the idea of functions prior to your reply. It just doesn't work for us because our loggers don't work well with functions and don't like the idea of not being able to control the number of instances. What we did was just to move the smaller WJ into another service plan – gris Dec 25 '20 at 16:32