Supervisor fails to restart half of the time

Question

I'm trying to deploy a Django app using Uwsgi and supervisor on a machine running Debian 8.1.

When I restart via sudo systemctl restart supervisor it fails to restart half of the time.

$ root@host:/# systemctl start supervisor
    Job for supervisor.service failed. See 'systemctl status supervisor.service' and 'journalctl -xn' for details.
$ root@host:/# systemctl status supervisor.service
    ● supervisor.service - LSB: Start/stop supervisor
       Loaded: loaded (/etc/init.d/supervisor)
       Active: failed (Result: exit-code) since Wed 2015-09-23 11:12:01 UTC; 16s ago
      Process: 21505 ExecStop=/etc/init.d/supervisor stop (code=exited, status=0/SUCCESS)
      Process: 21511 ExecStart=/etc/init.d/supervisor start (code=exited, status=1/FAILURE)
    Sep 23 11:12:01 host supervisor[21511]: Starting supervisor:
    Sep 23 11:12:01 host systemd[1]: supervisor.service: control process exited, code=exited status=1
    Sep 23 11:12:01 host systemd[1]: Failed to start LSB: Start/stop supervisor.
    Sep 23 11:12:01 host systemd[1]: Unit supervisor.service entered failed state.

However there is nothing in the supervisor or uwsgi logs. Supervisor 3.0 is running with this configuration for uwsgi :

[program:uwsgi]
stopsignal=QUIT
command = uwsgi --ini uwsgi.ini
directory = /dir/
environment=ENVIRONMENT=STAGING
logfile-maxbytes = 300MB

stopsignal=QUIT has been added because UWSGI ignores the default signal (SIGTERM) on stop and gets killed brutally with SIGKILL leaving orphan workers.

Is there a way I could investigate what's happening ?

EDIT:

Tried as mnencia advised : /etc/init.d/supervisor stop && while /etc/init.d/supervisor status ; do sleep 1; done && /etc/init.d/supervisor start but it still fails half of the time.

 root@host:~# /etc/init.d/supervisor stop && while /etc/init.d/supervisor status ; do sleep 1; done && /etc/init.d/supervisor start
    [ ok ] Stopping supervisor (via systemctl): supervisor.service.
    ● supervisor.service - LSB: Start/stop supervisor
       Loaded: loaded (/etc/init.d/supervisor)
       Active: inactive (dead) since Tue 2015-11-24 13:04:32 UTC; 89ms ago
      Process: 23490 ExecStop=/etc/init.d/supervisor stop (code=exited, status=0/SUCCESS)
      Process: 23349 ExecStart=/etc/init.d/supervisor start (code=exited, status=0/SUCCESS)

    Nov 24 13:04:30 xxx supervisor[23349]: Starting supervisor: supervisord.
    Nov 24 13:04:30 xxx systemd[1]: Started LSB: Start/stop supervisor.
    Nov 24 13:04:32 xxx systemd[1]: Stopping LSB: Start/stop supervisor...
    Nov 24 13:04:32 xxx supervisor[23490]: Stopping supervisor: supervisord.
    Nov 24 13:04:32 xxx systemd[1]: Stopped LSB: Start/stop supervisor.
    [....] Starting supervisor (via systemctl): supervisor.serviceJob for supervisor.service failed. See 'systemctl status supervisor.service' and 'journalctl -xn' for details.
     failed!
    root@host:~# /etc/init.d/supervisor stop && while /etc/init.d/supervisor status ; do sleep 1; done && /etc/init.d/supervisor start
    [ ok ] Stopping supervisor (via systemctl): supervisor.service.
    ● supervisor.service - LSB: Start/stop supervisor
       Loaded: loaded (/etc/init.d/supervisor)
       Active: failed (Result: exit-code) since Tue 2015-11-24 13:04:32 UTC; 1s ago
      Process: 23490 ExecStop=/etc/init.d/supervisor stop (code=exited, status=0/SUCCESS)
      Process: 23526 ExecStart=/etc/init.d/supervisor start (code=exited, status=1/FAILURE)

Nov 24 13:04:32 xxx systemd[1]: supervisor.service: control process exited, code=exited status=1
Nov 24 13:04:32 xxx systemd[1]: Failed to start LSB: Start/stop supervisor.
Nov 24 13:04:32 xxx systemd[1]: Unit supervisor.service entered failed state.
Nov 24 13:04:32 xxx supervisor[23526]: Starting supervisor:
Nov 24 13:04:33 xxx systemd[1]: Stopped LSB: Start/stop supervisor.
[ ok ] Starting supervisor (via systemctl): supervisor.service.

Answer 1

This is not necessarily an error from supervisor. I see from your systemctl status output that supervisor is started through the sysv-init compatibility layer, so the failure could be in the /etc/init.d/supervisor script. It would explain the absence of errors in the supervisord logs.

To debug the init script, the easiest way is to add a set -x as first non-comment instruction in that file, and look in the journalctl output the trace of the script execution.

EDIT:

I've reproduced and debugged it on a test system with Debian Sid.

The issue is that the stop target of the supervisor init-script does not check if the daemon has been really terminated but only send a signal if the process exists. If the daemon process takes a while to shutdown, the subsequent start action will fail due to the dying daemon process, which is counted as already running.

I've opened a bug on Debian Bug Tracker: http://bugs.debian.org/805920

WORKAROUND:

You can workaround the issue with:

/etc/init.d/supervisor force-stop && \
/etc/init.d/supervisor stop && \
/etc/init.d/supervisor start

• force-stop will ensure the supervisord has been terminated (outside systemd).
• stop make sure systemd know it's terminated
• start starts it again

The stop after the force-stop is required otherwise systemd will ignore any subsequent start request. stop and start can be combined using restart, but here I've put both of them to show how it works.

Answer 2

I had this issue in ubuntu 14.04, tried the latest initd script from debian and @mnencia solution but they didn't worked for me. force-stop solution did not kill the program processes they just the kept running after supervisord was killed.

My solution was to patch the supervisord and start and restart parts of the initd script code I didn't want to guess a good DODTIME, I wanted it to be up as soon as the old supervisor master process dies, so I added a retry logic. Note that it's kinda verbose, but you can just remove the echo calls if you don't like that behavior and you can change the max reties (set to 20 here).

start)
    echo -n "Starting $DESC: "
    i=1
    until [ $i -ge 21 ]; do
        start-stop-daemon --start --quiet --pidfile $PIDFILE --startas $DAEMON -- $DAEMON_OPTS  && break
        echo -n -e "\nAlready running, old process still finishing? retrying ($i/20)..."
        let "i += 1"
        sleep 1
    done
sleep 1
    if running ; then
        echo "$NAME."
    else
        echo " ERROR."
    fi
;;
restart)
    echo -n "Restarting $DESC: "
    start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
    i=1
    until [ $i -ge 21 ]; do
        start-stop-daemon --start --quiet --pidfile $PIDFILE --startas $DAEMON -- $DAEMON_OPTS  && break
        echo -n -e "\nAlready running, old process still finishing? retrying ($i/20)..."
        let "i += 1"
        sleep 1
    done
    echo "$NAME."
    ;;

I also changed the hashbang (first line) so bash is used insted of sh, I wanted to use let

#! /bin/bash