5

I've seen this problem expressed a lot but I've yet to find a working solution.

In short, I periodically have a large batch of processing operations to be done. Each operation is handled by an Azure Function. Each operation makes calls to a database. If I have too many concurrent functions running at the same time, this overloads the database and I get timeout errors. So, I want to be able to limit the number of concurrent Azure Function calls that are run at a single time.

I've switched the function to be queue-triggered and tweaked the batchSize / newBatchThreshold / maxDequeueCount host.json settings in many ways based on what I've seen online. I've also set the WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT application setting to 1 in my function application settings to prevent more than one VM from being spawned.

Yet still, every time I fill that queue, multiple functions will spawn indiscriminately and my database will fall over.

How can I throttle the number of concurrent operations?

vargonian
  • 3,064
  • 3
  • 27
  • 36
  • Based on the documentation there is no concurrent request handling for queue trigger. "WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT" this is in preview mode in V2 and currently there is no info on the field . HttpTrigger supports max concurrent request. https://github.com/Azure/Azure-Functions/issues/523 – Baskar Rao Nov 02 '18 at 21:12
  • 1
    @vargonian - what exactly are your configurations? Have you the combination of batchSize = 1 and WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1? – Marie Hoeger Nov 02 '18 at 21:59
  • @MarieHoeger Yes I've tried that combination, and it looks like maybe it's partially working insomuch as it's not scaling out (I'm guessing), but it still creates as many Azure Functions instances as one VM can handle because the trigger won't wait for a function instance to complete before pulling another item off the queue. So I can't gate it to just one at a time, for example. I'm hoping there's a way to do this. – vargonian Nov 03 '18 at 03:59
  • To limit the outgoing connections from one FunctionApp instance you can use a SemaphoreSlim as a static member (https://learn.microsoft.com/de-de/dotnet/api/system.threading.semaphoreslim?redirectedfrom=MSDN&view=netframework-4.7.2) – Sebastian Achatz Nov 03 '18 at 10:15
  • @vargonian - Hmm, ok either the host.json value is not getting honored or the app setting for scale-out is not getting honored. Each VM contains one instance of a function app, but function executions may occur concurrently if the host.json value is not being honored. Are you seeing concurrent function execution when running locally as well? – Marie Hoeger Nov 05 '18 at 18:53
  • @vargonian -- can you share what your host.json looks like? Setting batchSize to 1 should limit the concurrent per-VM functions to 1. – brettsam Nov 06 '18 at 20:11
  • @MarieHoeger I've yet to get a local testing environment set up but I will let you know what I see when I do that. My host.json currently looks like this: { "version": "2.0", "queues": { "batchSize": 1, "newBatchThreshold": 0, "maxDequeueCount": 1, "maxPollingInterval": 15000, "visibilitytimeout": "00:00:30" } } – vargonian Nov 06 '18 at 21:08
  • @vargonian - it looks like you're using host.json configurations from v1.x - there can you a host.json like this: https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-queue#hostjson-settings – Marie Hoeger Nov 07 '18 at 19:26
  • Try { "version":"2.0", "extensions": { "queues": { "maxPollingInterval": "00:00:02", "visibilityTimeout" : "00:00:30", "batchSize": 1, "maxDequeueCount": 1, "newBatchThreshold": 0 } } }, making sure that your `Microsoft.Azure.WebJobs.Extensions.Storage` is at least 3.0.1 – Marie Hoeger Nov 07 '18 at 22:27
  • @MarieHoeger Wow, this seems to have solved the problem, thank you! I double-checked at https://learn.microsoft.com/en-us/azure/azure-functions/functions-host-json and sure enough, I wasn't using that new format. Thank you again, this will really make a big impact. – vargonian Nov 08 '18 at 01:27

1 Answers1

10

The problem ended up being a difference in formatting of host.json in V1 and V2 Functions. Below is the correct configuration (using Microsoft.Azure.WebJobs.Extensions.Storage at least 3.0.1). The following host.json configures a single function app to process queue messages sequentially.

{ 
  "version":"2.0",
  "extensions": { 
    "queues": { 
      "batchSize": 1,
      "newBatchThreshold": 0 
     }
   } 
}

Setting App Setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1 restricts a function app from dynamically scaling out beyond one instance.

Marie Hoeger
  • 1,261
  • 9
  • 11
  • Thank you again for this solution. It seems that this host.json is a per-function-app setting rather than a per-function setting. Is this correct? I may have a situation in which I want different batch sizes depending on the function. – vargonian Nov 08 '18 at 17:54
  • Yes, host.json is per-function-app. I think the simplest and best thing to do here is to separate functions out to different function apps by required host configuration (the potential downside being additional management overhead)! – Marie Hoeger Nov 08 '18 at 18:41