1

I have 2 web servers (IIS 8.5) behind a hardware firewall and our application uses SignalR for some real-time updates. We are using SQL Server as the backplane to help us work in this load balanced environment. Additionally we are using sticky sessions on the load balancer to help us keep the users on the same web server during their session. When we are running in this hardware configuration we lose at least 1/3 of our messages. Sometimes we get all the expected messages but more often than not we are missing plenty.

When we are running on a single web server all messages are received. Does anyone have any suggestions for troubleshooting this problem? We've turned on logs (both client & server) and nothing looks like it's missing or broken. We're really stumped.

EDIT---

Some additional details that I hope will shed light on the situation.

  • Server to Client messages are getting lost. Pretty much all our communication is Server to Client.
  • We are using sticky session just based on IP and limited to 5 minutes but we're losing messages within that 5 minutes.
  • This is some old SignalR code that has been only minimally touched since SignalR 1 (or even older). We are keeping an in memory list of users along with their connections and we use that list to send notices back to the client. It seems most likely that this is the cause of the troubles but with Sticky sessions the user should be stuck to the same server for at least the 5 minutes right?
  • This list of users maps Username to connection id. This is useful when our backend services (on another machine) sends a message back with the username not the connection id.
Steven
  • 860
  • 6
  • 24
  • could you try keeping this list of users in a distributed memory, if not for good at least to narrow down on the problem. since you are already have sql server for signalr you could back your session in sql server and keep the list there if it is not too large. alternatively you could look at using a distributed cache like memcached or redis (or if that is too much then simply store in your own db table) – dove Oct 31 '17 at 08:26

2 Answers2

3

Finally resolved this. There were 2 issues really. The first is that we were using an in memory list of users as mentioned in the edit above. Once we realized that wasn't going to work across machines we removed it. It also led us to the second issue which was how SignalR 2 uses the IUserIdProvider and our call should have been 

Clients.User(userId).send(message)

instead of 

context.Clients.Client(connection)

This code had existed since we first started using SignalR many years ago and never got properly updated as we upgraded SignalR versions

Steven
  • 860
  • 6
  • 24
0

Have the same machineKey specified in your web.config on both servers.

dove
  • 20,469
  • 14
  • 82
  • 108
  • Already done. It hasn't made a difference. In our previous environment we used ARR as the load balancer and had to make that change. The previous environment which more consistent now with a hardware load balancer we're seeing consistently dropped messages. – Steven Oct 30 '17 at 12:58
  • @Steven I'm using Signalr behind hardware load balancers, with sql backplane too. I am also using sticky sessions. Is it client to client, client to server or server to client messages that are being lost? – dove Oct 30 '17 at 13:21
  • I updated the question with some additional details. – Steven Oct 30 '17 at 16:02