0

I have a serious issue with Azure Functions. Let me sketch my scenario.

I have a workload for processing incoming data files in a blob container. Using a BlobTrigger my processing function picks up the file, processes it and removes it from the container. Everything for one single file within 2 minutes of processing on the consumption plan.

Everything ok there, so I'm happy. Now I want to scale things up and offer +/- 10 of these batches in the blob container. The runtime starts executing them in parallel, so still works as expected. But then: the parallel executions of these functions slow down apparently, so the processing time of a single workload exceeds the max. execution time of 10 minutes! That's (more than) a factor 5 slowing things down, dependent on the amount of jobs offered.

Am I missing the point here? This is what the functions were built for or is it just some toy or something? I resorted to annotating my function with [Singleton] for now, which completely discards any scaling benefits here? How on earth could it be that 'serverless' execution units interfere with one another?

I am very puzzled, any insights would be greatly appreciated.

André Boonzaaijer
  • 764
  • 4
  • 13

1 Answers1

0

You should look into Azure function scaling algorithm

First-time Azure function allocates small instance (1.5 GB) to process workload, It would take some time to understand scaling requirement then only Azure ScaleController would add/remove the instance. For single blob 1.5 GB instance takes 2 mins, with 10 blob same instance (1.5 GB) would share resources and it would take time.

I don't see batch size setting for blob trigger in host.josn but I tried small batch size and solve a similar issue with ServiceBus.

You can send more workload on blob and with the help of application Insight, you can check Azure function scale out time.

I hope you already checked blob trigger issues.

Pankaj Rawat
  • 4,037
  • 6
  • 41
  • 73
  • Thanks for your answer. I've tried everything regarding these settings: - Blobtriggers cannot be configured like this, so I reverted to Queueus - Using 1 and 1 for batchSize & newBatch settings, yields exactly the same behaviour as using singleton. No scaling whatsoever, only one instance keeps (slowly) eating my messages. This really seems very immature. Anyone ever used this in a real world scenario? Or is it only demo-ware? – André Boonzaaijer Jan 14 '19 at 13:32
  • I'm using Azure function EventHub trigger and receiving thousands of messages in an hour. I can see multiple instance Id in Application Insights. – Pankaj Rawat Jan 15 '19 at 11:53
  • this is one of best article in Azure function scaling. https://mikhail.io/2018/11/from-0-to-1000-instances-how-serverless-providers-scale-queue-processing/ – Pankaj Rawat Jan 15 '19 at 11:54