Using (socket.io + RedisStore) to communicate across multiple servers

Question

I am working on a multiplayer online game, using Node.js and Socket.io. I expect a lot of players to join the game, so I am hosting it on Amazon Opworks.

The problem is that the servers aren't able to send socket events to clients connected to a different server. I am using RedisStore to manage socket.io sessions. I believed RedisStore and socket.io took care of this inter-server communication under the hood in a seamless manner. Here is a reference to another question: How does socket.io send messages across multiple servers?

But that's not the case it seems. Messages do not go through to other clients if they are on different servers; the app works if there is only one server, but fails if I use multiple servers loadbalanced using ELB on Opsworks.

This is just an extract from the whole code. Please ignore syntax errors etc if any.

app.js

    //Redis Client Initialization
   var redis = require("redis");
   redis_client = require('redis-url').connect('redis://xxxxx');

//setting RedisStore for socket.io 

var RedisStore = require('socket.io/lib/stores/redis')
, redis  = require('socket.io/node_modules/redis')
, pub    = redis.createClient(11042,'-----')
, sub    = redis.createClient(11042,'-----')
, client = redis.createClient(11042,'-----');

// using RedisStore in combo with Socket.io to allow communications across multiple servers

io.set('store', new RedisStore({
  redis    : redis,
  redisPub : pub,
  redisSub : sub,
  redisClient : client
}));

//socket communication specific code 


io.of('/game')
.on('connection', function (socket) {

  socket.on('init' , function(data){

    var user_id = data.user_id; // collecting user_id that was sent by the client
    var socket_id = socket.id;
    redis_client.set("user_socket:"+user_id, socket_id, function(err, reply){

          //stored a referece to the socket id for that user in the redis database

    });

  });

  socket.on('send_message', function(data){

      var sender = data.sender_id;

      var reciepient = data.reciepient_id ; // id of the user to whom message is to be sent to
      redis_client.get("user_socket:"+reciepient, function(err,socket_id){

        if(socket_id){

            var socket = io.of('/game').sockets[socket_id];
            socket.emit("message", {sender : sender}); // This fails. Messages to others servers dont go through.

        }
      })

  })

});

If I do a keys * , I do not see any message related data stored there.How do I inspect if the messaging is happening from Redis side ? — geeky_monster, Apr 29 '14 at 03:35
Are each of your servers pointing to the same redis server? Or are they each hosting their own redis? — clay, Apr 29 '14 at 17:42
I recommend tweaking your Opsworks configuration to first confirm you have Redis connectivity from a single instance in the most basic fashion, then be absolutely sure that you have a true "system of record" a single Redis database managing sessions across many servers. — Charney Kaye, May 01 '14 at 03:25
If I use just a single instance on Opsworks, everything works. So redis definitely works for Single instance. If I use multiple instance, Login and user auth works across all servers. Hence Redis connectivity works across all servers. — geeky_monster, May 01 '14 at 07:11
Can you please help elaborate on how to test the 'system of record' ? — geeky_monster, May 01 '14 at 07:15
facing exactly same problem. work on one instance and fail on multiple. have you found a solution? — mkto, Dec 23 '14 at 17:41
@mkto - Make use of rooms and broadcast to those rooms. Basically a pub-sub logic. — geeky_monster, Dec 24 '14 at 01:13
Hi thanks, what about room members? if let's say one member left a room on one instance, will another instance magically gets the updated room members when the code is loop through the clients list? — mkto, Dec 25 '14 at 02:41

score 1 · Accepted Answer · answered Oct 02 '14 at 05:28

You can't broadcast directly to socket objects on other servers. What Redis does is allows you to broadcast to 'rooms' on other servers. Thankfully with socket.io 1.x, new connections automatically join a room with the name of their socket id. To solve your problem, change:

    if(socket_id){

        var socket = io.of('/game').sockets[socket_id];
        socket.emit("message", {sender : sender}); // This fails. Messages to others servers dont go through.

    }

to emit to the room instead of calling emit on a socket object:

   if(socket_id){

        io.to(socket_id).emit("message", {sender : sender}); // This fails. Messages to others servers dont go through.

    }

And you might have more luck.

this information should be in the docs. socket.io people always leave out critical information — r3wt, Jun 30 '16 at 21:02
@geeky_monster did your redis + many-gameserver implementation, with the above solution, prove successful performance wise? thank you in advance — PathToLife, Jul 17 '21 at 10:16

Using (socket.io + RedisStore) to communicate across multiple servers

app.js

1 Answers1