Using Node cluster module with SailsJs: EADDRINUSE

Question

I have a SailsJs (http://sailsjs.org/) based application that has to deal with some CPU intensive tasks. In short, I want to use the cluster (https://nodejs.org/api/cluster.html) module to delegate the processing of these tasks to worker processes so that the main event loop of the Sails application isn't blocked (and so can respond to requests as normal).

When creating a worker, I'm getting an EADDRINUSE error as Sails is trying to run again and bind to the same port.

Sample code:

// SomeSailsService.js

var cluster = require('cluster');
var Queue = require('bull');
var myQueue = Queue('myQueue', 'connection stuff', 'etc');
var numWorkers = 2;
var i;

if (cluster.isMaster) {
  // Spawn some workers
  for (i = 0; i < numWorkers; i++) {
    cluster.fork();
  }
} else {
  // This is a worker, so bind to new job event
  myQueue.process(function processJob(job, done) {
    // CPU intensive shenanigans
  });
}

module.exports = {
  addToQueue: function(foo) {
    myQueue.add({foo: foo});
  }
};

When running the above, the Sails app starts up, then on starting the two workers tries to start the app another two times. This results in two of the following errors:

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: bind EADDRINUSE
    at errnoException (net.js:904:11)
    at net.js:1084:30
    at Object.1:1 (cluster.js:594:5)
    at handleResponse (cluster.js:171:41)
    at respond (cluster.js:192:5)
    at handleMessage (cluster.js:202:5)
    at process.emit (events.js:117:20)
    at handleMessage (child_process.js:322:10)
    at child_process.js:396:7
    at process.handleConversion.net.Native.got (child_process.js:91:7)
    at process.<anonymous> (child_process.js:395:13)
    at process.emit (events.js:117:20)
    at handleMessage (child_process.js:322:10)
    at Pipe.channel.onread (child_process.js:347:11)

Is there a way around this? Or an alternative way of tackling this problem?

Looks like you've figured it out, but if you'd like to chat more about this, come hang out in the sails.js Gitter room: https://gitter.im/balderdashy/sails — Travis Webb, Mar 15 '15 at 17:29

Tom Seldon · Answer 1 · 2015-03-17T00:41:19.047

I ended up following the advice of this blog post to boot up barebones instances of the app. Following on from that, the basic process to solve my problem was:

In app.js, check if cluster.isMaster === true.
If it is, launch the app as normal and create worker process(es).
If not, it's a worker process so boot up a minimal version of the app without the HTTP server. Register job process handlers.

The above blog post goes into more detail (including quite a nice implementation) of how to manage the creation and registering of these process handlers. It doesn't mention dealing with cluster, but hopefully the above steps help anyone else who has run into this problem.

EDIT:

On advice of Travis (see below comment), I dropped the use of the cluster module and instead created a worker.js file that did much the same thing. i.e. in worker.js I booted up a barebones Sails app started listening for new jobs. My main Sails app (the web API) is then just responsive for adding and getting jobs (low latency stuff).

Works nicely and should work easily with services such as Heroku.

Typically I would recommend just spinning up separate node instances, and use redis as a shared session store to share state between the nodes. This works basically out of the box on services like Heroku. — Travis Webb, Mar 15 '15 at 00:40
Tom! I just discovered this question, could you please be a saint and answer a question, I'm kinda stuck really badly: http://stackoverflow.com/questions/31866141/how-do-i-give-my-worker-js-access-to-the-sails-object — Amin Shah Gilani, Aug 06 '15 at 21:40

Using Node cluster module with SailsJs: EADDRINUSE

1 Answers1

Linked