Service Bus Timeout Exception

Question

We have a service bus topic with 300+ number of subscriptions, all these subscribers are azure functions (python) and we are facing a timeout exception that we are trying to understand better and is where we would appreciate inputs from the community please. 

Currently the timeout is set to 5 minutes and we are on consumption plan. Sharing below the configuration details for function app (host.json) along with the call stack for the exception. Any help on how to counter this would be appreciated. 

{
    "version": "2.0",
    "extensions": {
        "serviceBus": {
            "batchOptions": {
                "maxMessageCount": 10000,
                "operationTimeout": "00:05:00",
                "autoComplete": true
            },
            "prefetchCount": 0
        }
    },
    "logging": {
        "fileLoggingMode": "always",
        "logLevel": {
            "default": "Information",
            "Host.Results": "Information",
            "Function": "Information",
            "Host.Aggregator": "Information"
        },
        "applicationInsights": {
            "samplingSettings": {
                "isEnabled": false
            }
        }
    },
    "extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[3.*, 4.0.0)"
    },
    "concurrency": {
        "dynamicConcurrencyEnabled": true,
        "snapshotPersistenceEnabled": true
    }
}

Microsoft.Azure.WebJobs.Host.FunctionTimeoutException:
   at Microsoft.Azure.WebJobs.ServiceBus.MessageProcessor.CompleteProcessingMessageAsync (Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.8.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener+<ProcessMessageAsync>d__27.MoveNext (Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.8.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Azure.Messaging.ServiceBus.ServiceBusProcessor+<OnProcessMessageAsync>d__104.MoveNext (Azure.Messaging.ServiceBus, Version=7.11.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Azure.Messaging.ServiceBus.ReceiverManager+<OnMessageHandler>d__19.MoveNext (Azure.Messaging.ServiceBus, Version=7.11.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Azure.Messaging.ServiceBus.ReceiverManager+<ProcessOneMessage>d__15.MoveNext (Azure.Messaging.ServiceBus, Version=7.11.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)

Answer 1

Service Bus Timeout Exception:

Timeout exceptions occur for the following reasons:

As mentioned here in MSDoc, After the trigger starts function execution, the function needs to return/respond within the timeout duration.

Reason-1:-

• The timeout error is caused by the operation timeout being set to 05 minutes. If the batch processing takes longer than 05 minutes, a timeout error will be thrown.

Set the operation timeout to 10 minutes or more depending on the process running to resolve this.

Reason-2:-

• And I observed that the maximum message count was set to 10000.

Reduce the message count to a lower value to reduce the processing time of each batch.

I changed host.json as shown below:

"batchOptions": {
                "maxMessageCount": 5000,
                "operationTimeout": "00:10:00",
                "autoComplete": true
            }

Reason-3:-

Scale up the capacity of an app service plan that binds to the function app by changing the price tier and select apply as detailed in MSDoc.

Once it's done, you can again scale out the resources which are not in use.

Check this MSDoc for more occurrences and their solving abilities to avoid long time running of the functions.