3

I have an Azure Functions QueueTrigger that listens on a storage queue for messages like this:

Message text
--------------------------
{"ClientName": "client1"}
{"ClientName": "client2"}
{"ClientName": "client3"}

The QueueTrigger then has code like this:

if 'client1' == queue_msg['ClientName']:
    # do work required for client1
elif 'client2' == queue_msg['ClientName']:
    # do work required for client2
elif 'client3' == queue_msg['ClientName']:
    # do work required for client3

I'm using the Linux Consumption Plan with a batchSize of 1 because each invocation of the queue trigger can take about 5 minutes and I want to make sure I don't exceed memory limitations. This solution is working well for me now, but I'm concerned that when the amount of clients increases, messages will start to accumulate in the queue. Is it okay to just create a new Azure Function that also listens on the same storage queue? I think it would be okay because each message/client has work that is independent to them so it wouldn't matter if either of the Azure Function apps picked up a message first. This seems like the most cost effective solution for me, but I would like to know if there are better alternatives or any negative outcomes that I'm not thinking of.

ddx
  • 469
  • 2
  • 9
  • 27

1 Answers1

1

According to the description of your problem, you do not need to concern about the message accumulate. And you do not need to create another function to listen on the same storage queue. If you use consumption plan for your azure function, it will scale out more instances to deal with the messages. You can refer to this document. enter image description here enter image description here

===============================Update================================

According to your requirement of each instance running may use up to 1 GB memory, I suggest you to use "Premium plan" for your function.

When you create function app with premium plan, you can choose "EP1" as below screenshot. "EP1" plan has max 3.5GB memory, so it can scale out to 3 instances when the function running. enter image description here

After the function app created, go to the function app and click "Scale out(App Service Plan)", then set "Maximum Burst" as 3. It means the plan will most scale out to 3 instances. enter image description here

If you want more instances to run at same time, you can also choose "EP2" or "EP3" premium plan for your function app and change the value of "Maximum Burst".

Hury Shen
  • 14,948
  • 1
  • 9
  • 18
  • The reason I don't want to scale out is because the application can take about 5 minutes and can use up to about 1 GB of memory. If I have multiple instances running at the same time I'll get failures because I'll exceed the 1.5GB memory limit on the consumption plan. I've set `batchSize` to `1` and the `functionAppScaleLimit` to `1` because I don't want the app to scale out. That's why I was thinking creating another Azure Function would help process the messages. – ddx Nov 06 '20 at 06:07
  • Hi @JohnT Sorry I don't know each instance of your function can use up to 1 GB memory. But I have some doubts about that the failure of your function is caused by multiple instances in consumption. Because the [document](https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#service-limits) shows us max memory 1.5GB for each instance(not 1.5GB for all of instances). – Hury Shen Nov 06 '20 at 06:20
  • @JohnT Sorry, I review the document, it seems all of the memory is 1.5 GB. I will modify my solution later. – Hury Shen Nov 06 '20 at 06:28
  • Hi @JohnT Please refer to the "Update" of my answer. I suggest you to use "Premium plan" and the "Update" describes the plan in detail. – Hury Shen Nov 06 '20 at 06:43
  • Thanks @Hury Shen. It just seems like creating an additional Azure Function on the consumption plan would be more cost effective than upgrading to the premium plan. I’m not hitting any failures at the moment, I’m just trying to prepare for the future – ddx Nov 06 '20 at 09:14
  • Hi @JohnT The "Premuim plan" will not cost much more than consumption plan because it can also scale out. When it only use 1 instance or 0 instance, it will not cost much money. – Hury Shen Nov 06 '20 at 09:36
  • @JohnT If you do not have any other problem about this post, could you please mark my answer as "accepted", thanks in advance~ – Hury Shen Nov 06 '20 at 09:36
  • looking at the pricing page for Azure Functions https://azure.microsoft.com/en-us/pricing/details/functions/, it looks like the premium plan costs significantly more than the consumption plan. I don't understand why you think creating another Azure Function with the same QueueTrigger logic would be wrong. Is it possible for race conditions to occur? Can two messages be picked up at the same time by each QueueTrigger? I think my original question still has unanswered questions. – ddx Nov 06 '20 at 17:04
  • Hi @ddx I didn't say the solution to create another azure function queue trigger is wrong. I just meet your requirement `I would like to know if there are better alternatives` to suggest you with another solution with function host on premium plan because this solution do not need to create two function app with same code. Premium plan will cost more than consumption indeed, but I mean premium plan provides the same features and scaling mechanism used on the consumption plan which will save money for us(it is more cost-effective than other plan such as "app service plan"). – Hury Shen Nov 09 '20 at 01:57
  • @ddx For your question about `Can two messages be picked up at the same time by each QueueTrigger?` I think we do not need to worry about it. When function get one message from queue, the message will not be got by another function because the message will be locked(similar to locking mechanism of database). – Hury Shen Nov 09 '20 at 02:00
  • Thanks for the clarification @Hury Shen. For my use case, I think it would be more efficient and cost effective to create another Azure Function with the same queue code on the consumption plan. The premium plan might be a good solution temporarily, but I feel like I could control scaling better by creating separate functions and monitoring the queue. See this question for more details on my exact use case: https://stackoverflow.com/questions/64685629/how-to-make-sure-a-queue-message-has-been-successfully-processed-in-azure-functi/ I appreciate your help on this – ddx Nov 09 '20 at 04:05
  • if I upgrade to the premium plan and set `Maximum Burst` to `3`, will I have to change `batchSize` to `3` or can I leave it at `1` and the app will automatically scale to 3 concurrent instances? – ddx Nov 10 '20 at 19:42
  • 1
    @ddx I don't think we need to change `batchSize` from to `3`. You can refer to the [expression](https://i.stack.imgur.com/8IYnT.png) in this [document](https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-queue-output?tabs=csharp). – Hury Shen Nov 11 '20 at 06:50
  • If I'm on the Linux Consumption Plan and set the `functionAppScaleLimit` to 2 with the `batchSize` set to `1` does that mean two instances can run concurrently? Or would I need to be on the premium plan to with `Maximum Burst` set to `2` to achieve this? – ddx Nov 11 '20 at 17:26
  • 1
    Hi @ddx I will do some test and provide the answer under you new post. – Hury Shen Nov 12 '20 at 06:30