SignalR AspNetCore Unable to connect to the remote server ---> System.Net.WebException

Question

I am facing a problem while connecting to SignalRHub from my SignalR client application, below given is the error log-

info: Microsoft.AspNetCore.Sockets.Client.WebSocketsTransport[0]
      02/01/2018 15:20:13: Connection Id f763a939-3fb9-4812-ae6e-dfe3198ab37b: Starting transport. Transfer mode: Text.
fail: Microsoft.AspNetCore.Sockets.Client.HttpConnection[9]02/01/2018 15:20:13: Connection Id f763a939-3fb9-4812-ae6e-dfe3198ab37b: Failed to start connection. Error starting transport 'WebSocketsTransport'.
System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
   at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Sockets.Client.WebSocketsTransport.<Connect>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Sockets.Client.WebSocketsTransport.<StartAsync>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Sockets.Client.HttpConnection.<StartTransport>d__46.MoveNext()

SignalR hub is running with in stateless service fabric service and deployed on a azure service fabric cluster.

At the server side I am using nuget library Microsoft.AspNetCore.SignalR Below given is the AspNetCore 2.0 Stateless service side code for the reference-

Startup.cs:-

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

namespace SampleChat
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddMvc();
            services.AddSignalR();
            services.AddCors(options =>
            {
                options.AddPolicy("CorsPolicy",
                    builder => builder.AllowAnyOrigin()
                        .AllowAnyMethod()
                        .AllowAnyHeader()
                        .AllowCredentials());
            });
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            if (env.IsDevelopment())
                app.UseDeveloperExceptionPage();

            app.UseMvc();

            app.UseFileServer();

            app.UseCors("CorsPolicy");

            app.UseWebSockets();

            app.UseSignalR(routes => { routes.MapHub<ChatHub>("SignalRHub"); });
        }
    }
}

ChatHub.cs:-

using Microsoft.AspNetCore.SignalR;

namespace SampleChat
{
    public class ChatHub : Hub
    {
        public void Send(string message)
        {
            // Call the broadcastMessage method to update clients.
            Clients.All.InvokeAsync("Send", message);
        }
    }
}

SampleChat.cs

using System.Collections.Generic;
using System.Fabric;
using System.IO;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.ServiceFabric.Services.Communication.AspNetCore;
using Microsoft.ServiceFabric.Services.Communication.Runtime;
using Microsoft.ServiceFabric.Services.Runtime;

namespace SampleChat
{
    /// <summary>
    ///     The FabricRuntime creates an instance of this class for each service type instance.
    /// </summary>
    internal sealed class SampleChat : StatelessService
    {
        public SampleChat(StatelessServiceContext context)
            : base(context)
        {
        }

        /// <summary>
        ///     Optional override to create listeners (like tcp, http) for this service instance.
        /// </summary>
        /// <returns>The collection of listeners.</returns>
        protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
        {
            return new[]
            {
                new ServiceInstanceListener(serviceContext =>
                    new KestrelCommunicationListener(serviceContext, "ServiceEndpoint", (url, listener) =>
                    {
                        ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting Kestrel on {url}");

                        return new WebHostBuilder()
                            .UseKestrel()
                            .ConfigureServices(
                                services => services
                                    .AddSingleton(serviceContext))
                            .UseContentRoot(Directory.GetCurrentDirectory())
                            .UseStartup<Startup>()
                            .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                            .UseUrls(url)
                            .Build();
                    }))
            };
        }
    }
}

At the client side I am using nuget library Microsoft.AspNetCore.SignalR.Client Below given is the SignalR client side code-

Program.cs:-

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace SampleChatCoreApplication
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                SignalRConnector.ConnectoToSignalR().GetAwaiter().GetResult();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            };
            Console.ReadKey();
        }
    }
}

SignalRConnector.cs:-

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.AspNetCore.SignalR.Client;

namespace SampleChatCoreApplication
{
    static class SignalRConnector
    {
        public static async Task<bool> ConnectoToSignalR()
        {
            var connection = new HubConnectionBuilder()
                .WithUrl("http://localhost:8634/SignalRHub")
                .WithConsoleLogger()
                .WithTransport(Microsoft.AspNetCore.Sockets.TransportType.WebSockets)
                .Build();
            connection.On<string>("Send", data =>
            {
                Console.WriteLine($"Received data: {data}");
            });

            await connection.StartAsync();

            await connection.InvokeAsync("Send", "Send data to Hub");

            Console.ReadKey();

            return true;
        }
    }
}

I am able to connect with the SignalR Hub successfully when I run stateless service in local system.

I am not sure why I am facing the issue while connecting SignalR client with the SignalR hub when the service is running on the azure service fabric cluster.

For the additional information I have checked network and load-balancer rules and there is no connectivity issue from the network side. (I have verified it by adding a controller in this service and I am able to retrieve results from the controller).

Answer 1

I have did some R&D and found that issue can be related with the load balancer affinity. To verify Load Balancer Affinity issue I have deployed the application with HTTP protocol and I was able to connect with it in the multiple retry connection attempts. Which give me a hint that Load Balancer Affinity can cause the issue here. Further I have checked for the load balancer rule for the application port and found that load distribution was set to none. As per my understanding it should be "SourceIPProtocol". Currently the service is deployed in a 5 node service fabric cluster, when I scale down service to 1 node I was able to connect with the service in the first attempt via both HTTP and HTTPS protocol using a .net framework application.

Service is deployed using a self signed certificate.

Here the only issue left is that I am not able to connect with the service from a .net standard application via HTTPS protocol and getting the error- The certificate authority is invalid or incorrect

In order to connect from .net framework application I am writing below given line before the signalr connection code- ServicePointManager.ServerCertificateValidationCallback += (o, c, ch, er) => true;

Above given code was not working in the .net standard application.

By further R&D I found that in .NET Core, ServicePointManager was replaced with configuration on the HttpClientHandler instance itself. You need to call .WithMessageHandler on the IHubConnectionBuilder instance and provide an instance of HttpClientHandler with ServerCertificateCustomValidationCallback set.

I was not able to connect with the SignalR hub via HTTPS protocol when I used below given code-

var handler = new HttpClientHandler
                        {
                            ClientCertificateOptions = ClientCertificateOption.Manual,
                            ServerCertificateCustomValidationCallback = (httpRequestMessage, cert, cetChain, policyErrors) => { return true; }
                        };

                        Connection = new HubConnectionBuilder()
                            .WithUrl(url)
                            .WithTransport(Microsoft.AspNetCore.Sockets.TransportType.WebSockets)
                            .WithMessageHandler(handler)
                            .Build();

When I changed transport to TransportType.ServerSentEvents or TransportType.LongPolling, I was able to connect with the SignalR hub without any issue.