0

So, I will try and explain this to the best of my ability:

I wrote a TCP server that accepts data from a wearable (type of gps bracelet), using net in NodeJS. TCP server has connections = {} property, which contains all the connections, like this:

  • Server accepts connection
  • Device sends handshake packet, that contains device id
  • Socket is stored like: connections[device_id] = socket

Same TCP server also has express server which allows me to send data directly to the device using JSON post: curl -XPOST https://localhost:8080/send/device_id -d '{"command": "test"}'

Server parsers the url and writes to socket with connections[params.device_id].write()

This all works fine, when using pm2 start server.js. I am able to track devices, accept and send data, etc.

The issue is, when I start the server using pm2 start -i 5 server.js in cluster mode, device connections work okay, but express part is giving me issues.

For example: when I use curl -XPOST https://localhost:8080/send/device_id -d '{"command": "test"}' in 80% of the time, this will go to the wrong instance of server.js, because there are five copies of server.js running and each has express server, and since PM2 manages everything, my request will hit the tcp server where the given device_id is connect in 1 out of 5 times. The way I understand it, PM2 starts 5 instances of server.js and manages and balances the connections for tcp and express.

So, my question is, how should I manage this? Is there a way to tell PM2 to forward port 8080 to ALL running instances in a cluster? That way I could send single command to all instances, and in the instance, check if device_id is connected, and write to it's socket.

Is this possible, and if so, is it a good idea? If not, what is the right approach to this?

Hope I managed to explain

Regards

* EDIT *

It may not be clear from the initial question, but the socket in question is passed from net.createServer function, like

const net = require('net');
const PORT = 8181;
let connections = {}

const server = net.createServer((c) => {
  //... device id parse
  connections[device_id] = c;
});

server.listen(PORT, () => {
  console.log('Server started on', PORT);
});
dkasipovic
  • 5,930
  • 1
  • 19
  • 25
  • This sounds like you have 5 instances running and no sticky sessions so the user is not being directed back to the same server. Am I missing something or is there some load balancing in place? – Samuel Goldenbaum Dec 05 '19 at 23:41
  • @SamuelG I dont see where sticky sessions are applicable here? – dkasipovic Dec 06 '19 at 03:54
  • Are you saying that a user is being directed to different instances of server.js when they interacting with the API? – Samuel Goldenbaum Dec 06 '19 at 03:59
  • No, I am saying that the instance that handless the express request is not neccessarily an instance where socket connection is established (i.e. i have no way to ensuring that it is). Also there are no users here, it's only me interacting via curl – dkasipovic Dec 06 '19 at 04:47

1 Answers1

0

It's not clear if you plan to run all your server instances as workers on the same server, or may run multiple workers on multiple nodes.

The primary thing to change is your state storage within the instances. It sounds like you are storing data per worker by using calls like connections[params.device_id] but if these collections exist on in a single worker's memory, then there do not know about each other's data or other sockets that are connected.

If this is the case, then you need to move all your state management to a shared repository like Redis. Session data etc should move to a central repo too. It's best to architect the servers so that they are stateless if possible so you can spin up as many as needed.

Be sure your application is stateless meaning that no local data is stored in the process, for example sessions/websocket connections, session-memory and related. Use Redis, Mongo or other databases to share states between processes.

Socket.io has a redis-adapter to help facilitate communication between the socket.io instances so they are aware of socket connections and help facilitate broadcasting.

If you running on multiple servers, then you may need to have some sticky sessions in place so you are least being routed to the same server. But, if implement shared state suggested above, then it's not an issue. If that is not possible, then maybe some packages like express-sticky-cluster could help.

My own implementation allowed multiple workers on multiple nodes using shared data store and Redis backed adapter so that it didn't make a difference if the client reconnected to a different instance.

Hope that helps!

UPDATE There is even a guideline on PM2 specifying this exact architecture.

Samuel Goldenbaum
  • 18,391
  • 17
  • 66
  • 104
  • I am not sure how I can store socket connections inside redis. As I said `connection[device_id] = socket` where `socket` is a socket that is passed in onConnect function of `net` server – dkasipovic Dec 06 '19 at 09:18
  • This is not a unique scenario and is primarily how clustering works. You have tied your architecture to depend on the same worker always serving the same client, but now trying to run multiple workers in a load-balanced environment. PM2 even [highlights](https://pm2.keymetrics.io/docs/usage/cluster-mode/#statelessify-your-application) this. As per comments in the answer, you could try and implement sticky sessions for your scenario so you route source IP to the same worker similar to [this](https://github.com/elad/node-cluster-socket.io) – Samuel Goldenbaum Dec 06 '19 at 15:58