Upstart script for bitcoind, respawn feature

Question

I have an upstart script for the bitcoind, it based on the script in this topic: https://bitcointalk.org/index.php?topic=25518.0

I strongly need respawn future working: if something happened bitcoind should restart automatically. I tryed to emulate such situation, but upstart didn't restart the process.

Question: how can I make upstart (or something else) watch bitcoind and restart it if something bad happened?

Actual script:

description "bitcoind"

start on filesystem
stop on runlevel [!2345]
oom never
expect daemon
respawn
respawn limit 10 60 # 10 times in 60 seconds

script
user=root
home=/root/.bitcoin/
cmd=/usr/bin/bitcoind
pidfile=$home/bitcoind.pid
# Don't change anything below here unless you know what you're doing
[[ -e $pidfile && ! -d "/proc/$(cat $pidfile)" ]] && rm $pidfile
[[ -e $pidfile && "$(cat /proc/$(cat $pidfile)/cmdline)" != $cmd* ]] && rm $pidfile
exec start-stop-daemon --start -c $user --chdir $home --pidfile $pidfile --startas $cmd -b -m
end script

Could you explain how you emulated the situation? – CameronNemo Jun 12 '14 at 23:33 — CameronNemo, Jun 12 '14 at 23:33
I try to kill it with -15 and -9 signals – Aleksandr Chapkin Jun 14 '14 at 16:35 — Aleksandr Chapkin, Jun 14 '14 at 16:35

c.hill · Answer 1 · 2014-09-08T20:05:04.380

So I finally got things working on an Ubuntu 14.04 server. Here's what the final, working /etc/init/bitcoind.conf looks like:

description "bitcoind"

start on filesystem
stop on runlevel [!2345]
oom score -500
expect fork
respawn
respawn limit 10 60 # 10 times in 60 seconds

script
    user=bitcoind
    home=/home/$user
    cmd=$home/bin/bitcoind
    pidfile=$home/bitcoind.pid
    # Don't change anything below here unless you know what you're doing
    [[ -e $pidfile && ! -d "/proc/$(cat $pidfile)" ]] && rm $pidfile
    [[ -e $pidfile && "$(cat /proc/$(cat $pidfile)/cmdline)" != $cmd* ]] && rm $pidfile
    exec start-stop-daemon --start -c $user --chdir $home --pidfile $pidfile -m --startas $cmd
end script

Once you've added/updated your /etc/init/bitcoin.conf file, be sure to run the following:

initctl reload-configuration

Basically this was just a lot of guess and check to make this finally work. Here's the important bit:

expect fork

Essentially, this is telling upstart how many times the target process will be forked while starting. If you tell it wrong, it'll hang while starting. Read here for the specifics on this.

Also, the user I have bitcoind installed/running under is bitcoind instead of root.

You should be able to manually start bitcoind as a service, like this:

service bitcoind start

Or stop it, like this:

service bitcoind stop

If you restart your server, the bitcoind service should be started automatically. And, if the bitcoind process is killed or crashes, it will be automatically respawned. You can test that part out on your server by first finding the PID of the bitcoind process:

ps cax | grep bitcoind

Then, kill the process manually:

kill -9 PID_OF_BITCOIND

Then, try to get the PID of the bitcoind process again:

ps cax | grep bitcoind

It should still be running and with a new PID.

score 2 · Accepted Answer · answered Jun 12 '14 at 23:35

oom never

is your first problem. You need this:

oom score never

Additionally, do not use oom score never except for critical system services. Try -500 or -700 instead. That should be a higher priority than most processes, but not the ones that are essential for any running system. So you should use:

oom score -500

The second issue is that you are using start-stop-daemon. You should just ditch that, as Upstart can handle everything. So the resulting script would look like this:

description "bitcoind"

start on filesystem
stop on runlevel [!2345]

oom score -500
chdir /root/.bitcoin

respawn
respawn limit 10 60 # 10 times in 60 seconds

exec /usr/bin/bitcoind

The last issue could be that you have not defined normal exit properly. You need to specify which return codes and signals constitute a normal exit, so that Upstart knows to respawn if the signals and return codes do not match. See the Upstart cookbook on how to do this: http://upstart.ubuntu.com/cookbook/#normal-exit.

Thank you! It works with proper "normal exit" and "expect" variables. — Aleksandr Chapkin, Jun 14 '14 at 19:07

score 0 · Answer 3 · answered Jul 02 '14 at 12:12

So, to make upstart watch for the bitcoin and restart it if it falls I use

expect fork

Also, I didn't use start-stop-daemon and just run bitcoin with exec:

exec /path/to/bitcoind

And it's important to define normal exit codes (or code)

normal exit 0 15

Don't forget about respawn and respawn limit variables in your upstart config.

score 0 · Answer 4 · answered Nov 12 '14 at 13:51

0

There is now a pull request on the official bitcoin project which includes a better constructed upstart job

answered Nov 12 '14 at 13:51

George Angelopoulos

54
6

Upstart script for bitcoind, respawn feature

4 Answers4