19

I have an NodeJS application which sets up a UNIX-socket to expose some interprocess communication channel (some kind of monitoring stuff). UNIX-socket file is placed in os.tmpdir() folder (i.e. /tmp/app-monitor.sock).

var net = require('net');
var server = net.createServer(...);
server.listen('/tmp/app-monitor.sock', ...);

I use a signal handling (SIGINT, SITERM, etc...) to gracefully shutdown my server and remove a socket file.

function shutdown() {
    server.close(); // socket file is automatically removed here
    process.exit();
}

process.on('SIGINT', shutdown);
// and so on

My application is running with forever start ... to monitor it's lifecycle.

I have a problem with forever restartall command. When forever doing restartall it's using a SIGKILL to terminate all child processes. SIGKILL can't be handled by a process so my app dies without any shutdown procedures.

The problem is a socket file which is not removed when SIGKILL is used. After the child process is restarted, new server can't be started cause' a listen call will cause a EADDRINUSE error.

I can't remove a existing socket file during an app startup cause' I don't know if it a real working socket or some traces of a previous unclean shutdown.

So, the question is... What is the better way to handle such situation (SIGKILL and UNIX-socket server)?

Olegas
  • 10,349
  • 8
  • 51
  • 72
  • did u read this http://nodejs.org/api/all.html#all_signal_events – wayne Apr 24 '13 at 01:38
  • 5
    Yes, and did u read my question? – Olegas Apr 24 '13 at 04:36
  • no I did not read. It is easy to say than do. If you do not mind to modify the code in forever, then forever/node_modules/forever-monitor/lib/forever-monitor/monitor.js, in fuunction Monitor.prototype.kill add the SIGINT before forever send the SIGKILL signal – wayne Apr 24 '13 at 05:54
  • 3
    @wayne This is a bad solution. I don't want to modify 3rd party software. Also, my process may be killed with SIGKILL without forever... – Olegas Apr 24 '13 at 07:19
  • FYI [node-dev](https://github.com/fgnass/node-dev) sends a `SIGTERM` which is catchable. – mpen Apr 18 '17 at 21:20

5 Answers5

48

As other people have mentioned, you cannot do anything in response to SIGKILL, which is in general why forever (and everybody else) should not be using SIGKILL except in extreme circumstances. So the best you can do is clean up in another process.

I suggest you clean up on start. When you get EADDRINUSE, try to connect to the socket. If the socket connection succeeds, another server is running and so this instance should exit. If the connection fails then it is safe to unlink the socket file and create a new one.

var fs = require('fs');
var net = require('net');
var server = net.createServer(function(c) { //'connection' listener
    console.log('server connected');
    c.on('end', function() {
        console.log('server disconnected');
    });
    c.write('hello\r\n');
    c.pipe(c);
});

server.on('error', function (e) {
    if (e.code == 'EADDRINUSE') {
        var clientSocket = new net.Socket();
        clientSocket.on('error', function(e) { // handle error trying to talk to server
            if (e.code == 'ECONNREFUSED') {  // No other server listening
                fs.unlinkSync('/tmp/app-monitor.sock');
                server.listen('/tmp/app-monitor.sock', function() { //'listening' listener
                    console.log('server recovered');
                });
            }
        });
        clientSocket.connect({path: '/tmp/app-monitor.sock'}, function() { 
            console.log('Server running, giving up...');
            process.exit();
        });
    }
});

server.listen('/tmp/app-monitor.sock', function() { //'listening' listener
    console.log('server bound');
});
Old Pro
  • 24,624
  • 7
  • 58
  • 106
  • 1
    Great answer. Just one "error": when you call `fs.unlink` you should pass a callback, and restart the `server.listen` again on that callback. This way you never try to connect before unlinking (also node warns `fs: missing callback` when not using a callback). – Salvatorelab Mar 26 '14 at 10:42
  • 2
    @TheBronx please edit the answer to include your recommendation/improvement. – Old Pro Mar 28 '14 at 07:18
  • 1
    Slightly off-topic: It blows my mind how the whitespace edit from 2 spaces to 4 spaces was approved for this. – Claudiu Mar 28 '14 at 08:26
  • 1
    @Claudius: So true, I actually tried to reject it but was too late. – Stijn de Witt Mar 28 '14 at 08:27
  • It's a good thing Old Pro invited the edit. That makes up for a lot. – Stijn de Witt Mar 28 '14 at 08:28
  • That's how I got here, I was a few seconds late – Claudiu Mar 28 '14 at 08:28
  • Yup, it wasn't just whitespace :) – Claudiu Mar 28 '14 at 08:31
  • 1
    @TheBronx, Because this is a server startup issue with a practically unavoidable race condition, I want to use a synchronous unlink function to minimize the time between deleting the socket and creating a new one. I believe `fs.unlink` was synchronous when I wrote this (I certainly didn't get a warning when I ran it) but now that there is an explicitly synchronous version I replaced your change with that. – Old Pro Mar 30 '14 at 02:56
  • 1
    I wrote a simple npm package that implements this for easy usage: https://www.npmjs.com/package/server-starter – Cameron Tacklind May 28 '18 at 20:40
  • Here's a gist that gracefully starts the server for you, and deletes stale socket files: https://gist.github.com/ericvicenti/876e2df1103232f1fe6781c927e0eb63 – Eric Vicenti Dec 05 '18 at 21:08
2

you should be able to use SIGTERM to do what you want: process.on('SIGTERM', shutdown)

Clintm
  • 4,505
  • 3
  • 41
  • 54
  • I am able to handle `SIGTERM`, but `forever` restarts child processes by `SIGKILL` – Olegas May 08 '13 at 09:28
  • 3
    @Olegas - `forever` has an option to change the SIG used when it stops a process. You can change it to SIGTERM. – jfriend00 Sep 16 '14 at 07:21
1
server.on('error', function (e) {
  if (e.code == 'EADDRINUSE') {
    console.log('Address in use, retrying...');
    setTimeout(function () {
      server.close();
      server.listen(PORT, HOST);
    }, 1000);
  }
});

http://nodejs.org/api/net.html#net_server_listen_port_host_backlog_callback

UPD

you cant handle SIGKILL, then you must cleanup socket manual

this example work fine with forever

var fs = require('fs');
var net = require('net');
var server = net.createServer(function(c) {});
server.listen('./app-monitor.sock', function() {
  console.log('server bound');
});

server.on('error', function (e) {
  if (e.code == 'EADDRINUSE') {
    console.log('Address in use, retrying...');
    setTimeout(function () {
      fs.unlink('./app-monitor.sock');
    }, 1000);
  }
});
amirka
  • 313
  • 2
  • 12
  • But `EADDRINUSE` in case of file-socket server means that socket file exists. It won't ever close or disappear, cause' parent app is already dead, killed with `SIGKILL` – Olegas May 08 '13 at 09:31
  • SIGKILL close programm immediately use fs.unlink instead server.close() – amirka May 08 '13 at 13:03
  • ... and I'll potentially kill a socket from another running instance. – Olegas May 08 '13 at 18:10
  • 1
    UNIX sockets are file system entities, you need to clean them up after you're done. Make sure you call server.close() before you exit. if you can't do that, than you must fix it in forever-monitor to send another signal or in your application via fs.unlink – amirka May 09 '13 at 13:51
1

Since you cannot handle SIGKILL and you want to use forever (which uses SIGKILL) you will have to use a workaround.

For example, first send a signal to shutdown your server and then do the forever restart:

kill -s SIGUSR1 $pid
# $pid contains the pid of your node process, or use
# killall -SIGUSR1 node

sleep 2
forever restart test.js

In your js handle SIGUSR1:

process.on('SIGUSR1', gracefullyShutdownMyServer);
laktak
  • 57,064
  • 17
  • 134
  • 164
0

you must choose

  1. change SIGKILL to another signal in forever-monitor to handle in app
  2. take care of your app in this case, using fs.unlink
  3. stop using forever
amirka
  • 313
  • 2
  • 12