0

When enabling VNet integration on an Azure Function App to communicate with outbound internal services over private endpoints, an HTTP 502 Bad Gateway or HTTP 504 Gateway Timeout error occurs. If we disable VNet integration, the function app starts working normally.

Here is a diagram of our configuration:

Diagrams of Working and Not Working Architectures

In the logs, we see the following:

2022-10-14T15:04:59.443 [Information] Host initialized (1052ms)
2022-10-14T15:04:59.458 [Information] Host started (1107ms)
2022-10-14T15:04:59.459 [Information] Job host started
2022-10-14T15:04:59.500 [Debug] Debug file watch initialized.
2022-10-14T15:04:59.511 [Debug] Diagnostic file watch initialized.
2022-10-14T15:04:59.512 [Debug] Hosting started
2022-10-14T15:05:02.400 [Error] Unhandled exception. System.IO.IOException: The request was aborted.
2022-10-14T15:05:02.413 [Error] ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).
2022-10-14T15:05:02.418 [Error] at System.Net.Http.Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode)
2022-10-14T15:05:02.425 [Information] at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
2022-10-14T15:05:02.425 [Information] at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
2022-10-14T15:05:02.425 [Information] --- End of inner exception stack trace ---
2022-10-14T15:05:02.425 [Information] at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
2022-10-14T15:05:02.425 [Information] at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
2022-10-14T15:05:02.425 [Information] at Grpc.Net.Client.StreamExtensions.WriteMessageAsync[TMessage](Stream stream, GrpcCall call, TMessage message, Action`2 serializer, CallOptions callOptions)
2022-10-14T15:05:02.425 [Information] at Grpc.Net.Client.Internal.HttpContentClientStreamWriter`2.WriteAsyncCore[TState](Func`5 writeFunc, TState state)
2022-10-14T15:05:02.425 [Information] at Microsoft.Azure.Functions.Worker.GrpcWorker.SendStartStreamMessageAsync(IClientStreamWriter`1 requestStream) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 84
2022-10-14T15:05:02.426 [Information] at Microsoft.Azure.Functions.Worker.GrpcWorker.StartAsync(CancellationToken token) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 61
2022-10-14T15:05:02.426 [Information] at Microsoft.Azure.Functions.Worker.WorkerHostedService.StartAsync(CancellationToken cancellationToken) in D:\a\1\s\src\DotNetWorker.Core\WorkerHostedService.cs:line 25
2022-10-14T15:05:02.426 [Information] at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
2022-10-14T15:05:02.426 [Information] at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2022-10-14T15:05:02.426 [Information] at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2022-10-14T15:05:02.426 [Information] at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
2022-10-14T15:05:02.426 [Information] at MyCompany.MyApp.Api.Program.Main() in /home/vsts/work/1/s/src/Api/Program.cs:line 14
2022-10-14T15:05:02.681 [Debug] Handling WorkerErrorEvent for runtime:dotnet-isolated, workerId:dotnet-isolated. Failed with: Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException: dotnet exited with code 134 (0x86)
---> System.Exception: Unhandled exception. System.IO.IOException: The request was aborted., ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).,   at System.Net.Http.Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode)
   --- End of inner exception stack trace ---
2022-10-14T15:05:02.681 [Debug] Attempting to dispose webhost or jobhost channel for workerId: '6778e415-98a9-4167-8bf6-582416320e64', runtime: 'dotnet-isolated'
2022-10-14T15:05:02.681 [Debug] Disposing language worker channel with id:6778e415-98a9-4167-8bf6-582416320e64

What can cause this behavior? We have tried using the latest .NET Framework. There are also custom DNS servers and route tables configured in this environment, but we leverage the same infrastructure we use for all other private endpoint configurations, which work without issues.

Tomerikoo
  • 18,379
  • 16
  • 47
  • 61
HXK8
  • 41
  • 4
  • When you add private endpoint to azure function only the entities which are in the vnet where the private endpoint is setup will be able to access the function not any public ip address – Mohit Ganorkar Oct 15 '22 at 12:46
  • Thanks for responding @MohitGanorkar. The issue is not with inbound traffic to the private endpoint on the Azure Function App but with outbound traffic to private endpoints on other services (Azure Storage Accounts, Azure SQL Server). Or so I believe is the case. The only symptom we have is that once the Azure Function App is VNet integrated, attempting to access the function endpoints will lead to HTTP 502/504 errors and no invocation/logs are visible for those functions. – HXK8 Oct 16 '22 at 22:01
  • I was able to deploy and run a Dotnet isolated querytriggred function by first deploying the function then adding the private endpoint and vnet integration. Both of them are in same virtual network just in different subnet. Did you follow this process ? – Mohit Ganorkar Oct 25 '22 at 04:37
  • For us, we have an Application Gateway in front of the function app, so the inbound private endpoint doesn't change. The outbound private endpoints to Azure Storage Account and Azure SQL Server also remain static. The only change is VNet integration onto a subnet in the same VNet as all other private endpoints. The function is an HTTP-triggered .NET application instead of query-triggered, but that shouldn't matter much as the issue appears to be on startup/runtime rather than how it is triggered. – HXK8 Oct 31 '22 at 15:41

2 Answers2

1

I could solve the problem by removing .AddEnvironmentVariables() from ConfigureAppConfiguration()

Tomerikoo
  • 18,379
  • 16
  • 47
  • 61
ceekaay
  • 11
  • 1
0

The cause appears to be an unidentified and unresolved defect from the Azure App Service side. There is no known official solution. However, we were able to work around the solution by re-engineering the application to switch the .NET worker process from 'isolated' to 'in-process.' Here is a longer description from our development team:

Earlier, the dotnet core application (.Net 6) was running with runtime set to 'isolated', i.e., it was using Microsoft.Azure.Functions.Worker.* dependencies in the code. This was throwing runtime exception at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel whenever we hosted the function app to Azure.

We switched the runtime to 'in-proc', by removing all Microsoft.Azure.Functions.Worker.* dependencies - This involved some changes to the application code because things like middlewares were not automatically supported now. But removing those dependencies and changing runtime did the trick. We did not see any runtime errors in Azure.

Everything else like storage-account, platform (Linux) was kept same. Only change was in the application runtime and the Microsoft worker dependency.

Thanks.

HXK8
  • 41
  • 4