0

I have an Azure Function App inside an App Service Plan - Elastic Premium. The Function App is inside a VNet and it is a durable function app with some functions triggered by non-HTTP triggers:

enter image description here

I noticed the the Service Plan is not able to scale out properly, indeed the number of minimum instances is 4 and maximum instances 100

enter image description here

but checking the number of instances in the last 24 hours in:

"Function App -> Diagnose and solve problems -> Availability and performance -> HTTP Functions scaling"

the number of instances is never higher than 5/6 instances: enter image description here

which is not good because if I check in:

"Function App -> Diagnose and solve problems -> Availability and performance -> Linux Load Average"

I have the following message:

"High Load Average Detected. Shows Load Average information if system has Load Average > 4 times # of Cpu's in the timeframe. For more information see Additional Investigation notes below."

enter image description here

and also checking the CPU and memory usage from the metrics I have some high spikes

enter image description here

so this means that my App Service plan is not able to scale out properly. One of my colleague, suggested to verify that:

"Function App -> Configuration -> Function runtime settings -> Runtime Scale Monitoring"

is set to "on": enter image description here

because if it is set to "off", may be that the VNet blocks Azure from diagnostic our app and as a result, Azure is not spawning more instances because is not seeing in real-time what CPU Load is. But I didn't understand how this setting can help me to scale out. Do you know what the "Runtime Scale Monitoring" is used for and why this can help me to scale out?

And also, do you think is more a problem related to scaling up instead of scaling out?

Gregory
  • 75
  • 1
  • 7

2 Answers2

0

FUNCTIONS_WORKER_PROCESS_COUNT - This will Scale-up the Function Workers but not the hosts running.
It means if you set the value to X, then each host instance runs the X individual function worker processes concurrently.

This setting will scale up the host instance to run as many as Function Worker processes concurrently based on specified value.

Runtime Scale Monitoring requires the Contributor Access Level on both App Service Plan and Function App Level to enable the setting for Scaling Up/down operation performing.

enter image description here

I believe the above settings should need the pre-warmed instances to be at least set to 1.

For more information on how runtime scale controller works and cost optimization by enabling above settings, I have found the useful information in SO 1 and2 provided by the users @HariKrishna and @HuryShen.

Updated Answer:

In Azure Functions, an orchestrator function code can be either HTTP, Timer or any other Code Events, Sub-orchestrations, and activity functions. To deal with the multiple orchestrator functions in parallel, you can use the below setting under extensions > durableTask > maxConcurrentOrchestratorFunctions to X Value in the host.json file. Refer to the JonasW Blog Article regarding how scaling happens in Azure Durable Functions.

Pravallika KV
  • 2,415
  • 2
  • 2
  • 7
  • Sorry @Pravallika, so the FUNCTIONS_WORKER_PROCESS_COUNT is related to the scaling up process, I wasn't aware of this. Regarding instead the Runtime Scale Monitoring, is this a configuration related to scaling up or scaling out? Anyway I have edited my question to be more clear. – Gregory Feb 13 '23 at 14:02
  • 1
    Updated the answer with additional information regarding how azure durable functions will scale up. – Pravallika KV Feb 14 '23 at 06:38
0

I assume that you are not using HTTP triggers, but instead something like ServiceBus. Also I don't think there is any "scaling up" in Consumption or Premium plans.

https://learn.microsoft.com/en-us/azure/azure-functions/functions-networking-options?tabs=azure-cli#premium-plan-with-virtual-network-triggers

When you enable virtual network trigger support, only the trigger types shown in the previous table scale dynamically with your application. You can still use triggers that aren't in the table, but they're not scaled beyond their pre-warmed instance count. For the complete list of triggers, see Triggers and bindings.

My understanding is that the setting will allow the scale controller to gain insight into what is triggering your function. So it is not scaling horizontally by looking at CPU usage, but rather by looking at the executed triggers to check if the messages are being processed fast enough.

https://learn.microsoft.com/en-us/azure/azure-functions/event-driven-scaling#runtime-scaling

For example, when you're using an Azure Queue storage trigger, it scales based on the queue length and the age of the oldest queue message.

If you disable the setting, the Scale Controller shown in the image will not have access to the queue length and you will not observe any scaling.

scaling info

Alex AIT
  • 17,361
  • 3
  • 36
  • 73
  • Thank you @Alex, your answer is very useful! Indeed I don't have HTTP triggers, what I have is a durable function app with some functions that are triggered by non-HTTP triggers, like "Activity" or "Orchestration" trigger types (I have edited the original question to add this information, so it will be more clear for other readers). At the end I suppose that if I set Runtime Scale Monitoring to "on", my durable function app will scale out better since it uses some non-HTTP triggers, right? – Gregory Feb 13 '23 at 16:45
  • @Gregory correct – Alex AIT Feb 13 '23 at 17:12