Summary:
Azure SignalR service has problems calling our upstream URL with HTTPInvocationFailed error. This happens to both our dev and production deployments concurrently.
Details:
We have a an Azure SignalR Service deployed in serverless mode. We have an Upstream URL setup which subscribes to all the events. We then bind to these events using Function Triggers to execute our business logic.
Link Azure Article about this configuration
Every once in a while SignalR Service shows HttpInvocationFailed errors with "Failed to connect with egress" as the Exception. Since this binding is done via an extension deployed on Azure Function App, we have no further visibility into these issues. Sometimes we get lucky and this issue resolves itself. Sometimes, we change something unrelated on the service (not our code) and it works again. However, at times our only option is to open a ticket and wait for Azure support.
We have dev and prod deployments and this happens to both at the same exact time.
Why do use it this way?
We need this primarily for group join/leave functionality where we need to know the connectionid of the user. We cannot use user name because the end user can be connected via multiple connections to the service. Subscribing to the events gives us access to the connection context where we can get the connectionid.
What we have tried
- Removed any custom code from the trigger that handles the events and return Task.CompletedTask to remove any errors from our code. When this happens, the trigger is not called so that really doesn't solve the issue.
- Restarted the Azure SignalR service. Since this happened to both production and dev services at the same exact time, we thought this was Azure related.
- Scaled-up and Scaled out the service. Hoping that the service would be moved to different instances that don't have this possible reachability issue.
- Deleted and re-deployed function app.
- Created new function app and updated the upstream url.
Questions:
- Is anyone else using SignalR Serverless mode with an Upstream URL?
- Does anyone know how to troubleshoot this further? We have picked up the details from Live Trace and LogAnalytics but we don't see actual underlying reasons.
- Is there another way to get the connectionid? We worked on an insecure way to achieve this with direct Azure function calls but that is not going to work for us.
