1

Scenario

We use a Vagrant-based virtual machine running Ubuntu 12.04 LTS for development of three Node.JS servers that are started using foreverjs. We run the services under the vagrant user and mount the projects from the host to the virtual machine as shared folders.

When starting a script with forever, it will fork to the background after starting, maintaining a central list of all scripts running (per user). For scripts that use different names, you can reference them by name as well as a numeric index.

Scripts are then managed (started, restarted, stopped, etc) using the forever command.

Upstart configuration

I have created similar upstart conf files for each service thusly:

description "Control server.js"

chdir /vagrant/server

start on vagrant-mounted

stop on runlevel [016]

expect fork

pre-start script
  test -d /vagrant/server
end script

exec sudo su vagrant -c "/usr/local/bin/forever start server.js"

Problem — hangs on shutdown

The upstart successfully starts the service once the vagrant shared folder has mounted. However, shutting down the virtual machine (i.e., vagrant halt) hangs until it times out and force-shuts-down the VM.

I assume this is because it doesn't know how to stop the forever service, since forever is merely an interface to the services I actually want to stop.

My only attempt didn't work (it does stop the service, but the Vagrant hangs to timeout):

pre-stop script
  forever stop server.js
end script

The node services do not need to terminate gracefully: I would be happy with any scenario that kills the scripts but does not cause the Vagrant to hang on shutdown.

Also, removing the stop on runlevel line seems to have no effect.

msanford
  • 1,477
  • 15
  • 28
  • 1
    If you attempt to stop one of the jobs with `initctl stop -n myserver` then do `initctl status myserver`, do you see something along the lines of `myserver stop/killed, process xxx`? I think the `expect fork` stanza is suspect. – CameronNemo Jul 18 '14 at 16:08
  • @CameronNemo I suspected it, too, but was unsure how to fix it as it's a process manager that _itself_ forks after being run. I'll look at your solution below! – msanford Jul 18 '14 at 19:09

1 Answers1

1

First, your problem seems to be improper fork tracking. To solve this issue, I am going to make a few tweaks to your upstart config:

  • First, scratch the usage of su to change the user. su should not be used for daemons, only regular users (it can cause problems with pam being triggered), and (more relevantly) it forks the process to the background (harder for Upstart to track and kill your daemon). We will use setuid vagrant in the upstart config instead.

  • The next problem is that forever forks the process to the background and forks itself to the background. This leaves Upstart tracking some random pid that probably is not even alive anymore. The solution is to use Upstart's automatic respawning feature. I will demonstrate by example below.

    description "server.js"

start on vagrant-mounted
stop on runlevel [016]

chdir /vagrant/server
setuid vagrant

respawn
respawn limit unlimited

exec server.js

Note that you do not need to check if /vagrant/server is a directory because upstart automatically creates it because of the chdir stanza.

CameronNemo
  • 399
  • 1
  • 6
  • The `test -d` is to check that the shared folder has actually been mounted (`forever start` can not traverse the filesystem and for whatever reason must be run from the same location as the entry point, hence the `chdir`). I also don't want the process to respawn by itself if it dies, since these are development instances that might be killed on purpose. I will try this solution and see how it works, though! I didn't consider `setuid` which will go a long way to solving this. – msanford Jul 18 '14 at 19:07
  • If you want to be able to kill them, you can just use initctl, or add `normal exit 0 TERM` so that if you kill the process with that signal (which is the default for kill), it will not respawn. – CameronNemo Jul 19 '14 at 15:18
  • 1
    Oh I see about the test now, yes that is easy to add back in. – CameronNemo Jul 19 '14 at 15:21