MaxBatchSize in Host.json for Azure Function not working

Question

I have below confiugration for my Azure Function in host.json & I am sending 30000 msgs in multiple batches to EvenHub but I see that my azure function doesn't pick more than 64 which is a default value. Any particular reason that it doesn't pick the maxbtachsize value.

Event Hub configuration is with Partition Count 25 for single event Hub and Single TU as 15

"version": "2.0",
"extensions": {
    "eventHubs": {
      "batchCheckpointFrequency": 5,
      "eventProcessorOptions": {
        "maxBatchSize": 256,
        "prefetchCount": 512
      }
    }
}

score 2 · Answer 1 · answered Oct 03 '21 at 17:39

For dotnet core, I had this problem that application did not take the maxBatchSize or other params mentioned in host.json. I could not do it from host.json. But it is done using dependency injection (IOptions)

public override void Configure(IFunctionsHostBuilder builder)
{
    builder.Services.PostConfigure<EventHubOptions>(o =>
    {
        o.EventProcessorOptions.MaxBatchSize = 256;
        o.EventProcessorOptions.PrefetchCount = 512;
    });
}

Now, it works for me! It is drawing 256 events if available.

score 0 · Answer 2 · answered Dec 25 '19 at 02:08

You should not expect events to be delivered exactly in batch sizes as configured by maxBatchSize. maxBatchSize defines the maximum batch size so events can come in 1 to maxBatchSize batch sizes.

If a backlog is building due to single host not able to process much then Azure Functions should scale out by adding more hosts. When new hosts become active, they should take over some of the partitions thereby increasing the read and process rate. See scaling section here for more details -https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs

Thanks it makes sense.... I was expecting that when my load is high atleast I should see some of them taking 256 than 64 which I didn't see and hence was thinking that if it is working fine or not..... — Punit, Dec 27 '19 at 21:45

score 0 · Answer 3 · answered Jun 08 '21 at 08:57

"enableReceiverRuntimeMetric": true,

Gives you maxBatchSize count if events are available.

"eventHubs": {
            "batchCheckpointFrequency": 5,
            "eventProcessorOptions": {
                "enableReceiverRuntimeMetric": true,
                "invokeProcessorAfterReceiveTimeout": false,
                "maxBatchSize": 256,
                "prefetchCount": 512,
                "receiveTimeout": "00:05:00"
            }
        }

MaxBatchSize in Host.json for Azure Function not working

3 Answers3