0

I have a Function App with multiple queue triggers set up and am trying to limit concurrency by setting functionAppScaleLimit to 1. After testing this, I'm getting timeouts because two queue triggers will execute at around the same time, but only one is allowed to do work while the other one waits.

For example, I have two queue triggers: QueueTrigger1 and QueueTrigger2 that are executed when blobs are created in two separate places in Azure Storage. I only want one of the queue triggers to be able to run at a time. I've already set the batchSize parameter to 1 so only one message is processed at a time. Each queue trigger can take up to 8 minutes for the scripts to complete execution. If both triggers are executed at around the same time, one will complete and the other will time out and then retry with a successful completion.

Here's an example log from QueueTrigger1:
2020-10-26 07:37:49.201 Executing 'Functions.QueueTrigger1' (Reason='New queue message detected on 'etl-queue-items-1'.', Id=<queue-trigger-1-id>)
//processes work in Python
2020-10-26 07:45:49.472 Executed 'Functions.QueueTrigger1' (Succeeded, Id=<queue-trigger-1-id>, Duration=480291ms)

And QueueTrigger2:
2020-10-26 07:37:56.922 Executing 'Functions.QueueTrigger2' (Reason='New queue message detected on 'etl-queue-items-2'.', Id=<queue-trigger-2-id>)
//8 minutes later
2020-10-26 07:45:49.439 Python queue trigger function processed a queue item:
//attempts to process work in Python
2020-10-26 07:47:56.927 Timeout value of 00:10:00 exceeded by function 'Functions.QueueTrigger2' (Id: '<queue-trigger-2-id>'). Initiating cancellation.
2020-10-26 07:47:56.987 Executed '{functionName}' ({status}, Id={invocationId}, Duration={executionDuration}ms)
2020-10-26 07:47:56.987 Executed 'Functions.QueueTrigger2' (Failed, Id=<queue-trigger-2-id><queue-trigger-2-id>, Duration=600043ms)

It seems unfair that the 10 minute limit is being applied to QueueTrigger2 before it even starts doing any work. How can I ensure that each queue trigger runs independently so I can be sure not to exceed the 1.5GB memory limit and not have to rely on retries?

Edit: I'm on the Linux Consumption plan so my functionTimeout is limited to only 10 minutes.

ddx
  • 469
  • 2
  • 9
  • 27

1 Answers1

1

You have already set functionAppScaleLimit to 1, another thing you should do is setting the batch size to 1 in host.json file according to this document :

If you want to minimize parallel execution for queue-triggered functions in a function app, you can set the batch size to 1. This setting eliminates concurrency only so long as your function app runs on a single virtual machine (VM).

Carl Zhao
  • 8,543
  • 2
  • 11
  • 19