Unable to start a Python app using Supervisord

Question

I have a small app that I want to start using supervisord. I've tried the following

Installed supervisord and controlled a small script
Read Introduction, Running Supervisord and Configuration File sections from the documentation
Setting up groups in supervisord

My initial shell script could start and stop celery and Flask as daemons by saving the PID in a text file. Since supervisord would take care of killing it, I got rid of the stop section and non daemonized the script.

After trial and error these are the script and conf duos that I think make sense but they don't work.

1

Shell Script

#!/bin/bash


if [[ $1 == "gunicorn" ]]
then
    cd /home/abhirath/Desktop/Hitler
    source env/bin/activate
    python env/bin/gunicorn -b 0.0.0.0:3333 -w 3 gunicornserve:app

elif [[ $1 == "celery" ]]
then
    cd /home/abhirath/Desktop/Hitler
    source env/bin/activate
    python env/bin/celery -A testrunner worker --concurrency=3 --loglevel=info

else
    echo "Usage:-"
    echo "To start celery:-"
    echo "./hitler.sh celery"
    echo "To start Gunicorn"
    echo "./hitler.sh gunicorn"
fi

Conf File

[group:hitler]
programs=gunicorn,celery

[program:gunicorn]
command=/home/abhirath/Desktop/Hitler/hitler.sh gunicorn
stderr_logfile =/home/abhirath/Desktop/supervisor.err.log
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true

[program:celery]
command=/home/abhirath/Desktop/Hitler/hitler.sh celery
stderr_logfile=/home/abhirath/Desktop/supervisor.err2.log
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true

2

Without a shell file

[group:hitler]
programs=gunicorn,celery

[program:gunicorn]
command=source env/bin/activate; python env/bin/gunicorn -b 0.0.0.0:3333 -w 2 gunicornserve:app;
directory=/home/abhirath/Desktop/Hitler

[program:celery]
command=source env/bin/activate; python env/bin/celery -A testrunner worker --concurrency=4 --loglevel=info;
directory=/home/abhirath/Desktop/Hitler

stderr_logfile, autostart, autorestart, stopasgroup, killasgroup same as #1

I get a message saying could not find the command source. I tried the same command on the terminal in the same directory and it works.

3

Shell Script

#!/bin/bash
    
    
    if [[ $1 == "gunicorn" ]]
    then
        source env/bin/activate
        python env/bin/gunicorn -b 0.0.0.0:3333 -w 3 gunicornserve:app
    
    elif [[ $1 == "celery" ]]
    then
        source env/bin/activate
        python env/bin/celery -A testrunner worker --concurrency=3 --loglevel=info
    
    else
        echo "Usage:-"
        echo "To start celery:-"
        echo "./hitler.sh celery"
        echo "To start Gunicorn"
        echo "./hitler.sh gunicorn"
    fi

Conf

[group:hitler]
programs=gunicorn,celery

[program:gunicorn]
command=./hitler.sh gunicorn
directory=/home/abhirath/Desktop/Hitler    

[program:celery]
command=./hitler.sh celery
directory=/home/abhirath/Desktop/Hitler

stderr_logfile, autostart, autorestart, stopasgroup, killasgroup same as #1

I also tried using command=bash -c "command here" even though I feel it isn't required in the all the above cases. It's mentioned here in the documentation.

I get the following errors and I'm not able to figure out why:-

Could not spawn

Process Exited too quickly

In #2 you do not need **source**. You can use: command=/path/to/env/bin/gunicorn -b 0.0.0.0:3333 -w 2 gunicornserve:app — Amin Alaee, Nov 30 '16 at 07:32
Note for #3. According to [document](http://supervisord.org/configuration.html#program-x-section-values) "*If it is relative, the supervisord’s environment $PATH will be searched for the executable*", it means that `./hitler.sh` is searched from `$PATH`, not `/home/abhirath/Desktop/Hitler` — ymonad, Nov 30 '16 at 07:34
@ymonad I do not understand that part. I'll ask my colleagues to elaborate on that. Thanks :) — Abhirath Mahipal, Nov 30 '16 at 07:44
@MohammadAmin But I need to activate the Python Environment. Don't I? I'll try and get back to you :) — Abhirath Mahipal, Nov 30 '16 at 07:45
@Abhirath Mahipal This way you are using the virtualenv, you don't need to activate it. — Amin Alaee, Nov 30 '16 at 07:50
@AbhirathMahipal it means that you might have to add `environment = PATH="/home/abhirath/Desktop/Hitler:/usr/local/bin:/usr/bin:/bin"` if you want to set `command=./hitler.sh` — ymonad, Nov 30 '16 at 07:54
@ymonad Thanks. Please answer the question so that I can upvote it and probably accept it in case I don't get a better answer :) — Abhirath Mahipal, Nov 30 '16 at 07:58

Amin Alaee · Accepted Answer · 2016-11-30T12:25:52.327

2

In the case of #2, you don't actually need to activate the virtualenv. You can change it to this:

[group:hitler]
programs=gunicorn,celery

[program:gunicorn]
command=/absolute/path/to/env/bin/gunicorn /absolute/path/to/gunicornserve:app -b 0.0.0.0:3333 -w 2
directory=/home/abhirath/Desktop/Hitler

[program:celery]
command=/absolute/path/to/env/bin/celery -A testrunner worker --concurrency=4 --loglevel=info
directory=/home/abhirath/Desktop/Hitler

edited Nov 30 '16 at 12:25

answered Nov 30 '16 at 07:54

Amin Alaee

1,895
17
26

I'll just try it out. Can I wait for sometime even after it works before accepting your answer? – Abhirath Mahipal Nov 30 '16 at 07:57
Sure, but please choose an answer for your question so future visitors can make use of it too. Thanks – Amin Alaee Nov 30 '16 at 07:59
Yes. Absolutely :) – Abhirath Mahipal Nov 30 '16 at 08:00
I get `BACKOFF Exited too quickly`. It gives out could not spawn process and the statements like `could not find module` – Abhirath Mahipal Nov 30 '16 at 08:09
gunicorn and celery. I also tried it with `python /path/to/env/bin/gunicorn ...`. Don't bother yourself just yet, I'll try @ymonad's solution first. – Abhirath Mahipal Nov 30 '16 at 08:22
Just a quick reminder, you should change /path/to/env to your own environment path which would be something like /home/abhirath/Desktop/Hitler/env – Amin Alaee Nov 30 '16 at 08:26
Please edit your answer. `/absolute/path/to/env/python /absolute/path/togunicorn arguments here`. This worked for me. – Abhirath Mahipal Nov 30 '16 at 12:17
`command=/path/to/env/bin/python /path/to/env/bin/gunicorn arguments` `command=path/to/env/bin/python /path/to/env/bin/celery arguments` – Abhirath Mahipal Nov 30 '16 at 12:41
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/129439/discussion-between-mohammad-amin-and-abhirath-mahipal). – Amin Alaee Nov 30 '16 at 12:46

score 0 · Answer 2 · edited Feb 21 '18 at 13:40

@MohammadAmin and @ymonad gave wonderful suggestions which I used in this answer. I found a simpler solution due to the explanation given here -> Supervising virtualenv django app via supervisor

Script File

#!/bin/bash


if [[ $1 == "gunicorn" ]]
then
  env/bin/python env/bin/gunicorn -b 0.0.0.0:3333 -w 3 gunicornserve:app 

elif [[ $1 == "celery" ]]
then
  export C_FORCE_ROOT='true'
  env/bin/python env/bin/celery -A testrunner worker --concurrency=3 --loglevel=info

else
  echo "Usage:-"
  echo "To start celery:-"
  echo "./hitler.sh celery"
  echo "To start Gunicorn"
  echo "./hitler.sh gunicorn"

fi

Since I specify the project directory as an environment path, I needn't specify the absolute path to the Python interpreter in the environment.

Supervisord Conf

[group:hitler]
programs=gunicorn,celery

[program:gunicorn]
command=bash -c "./hitler.sh gunicorn" 
directory=/home/abhirath/Desktop/Hitler
environment=PATH="/home/abhirath/Desktop/Hitler"
stderr_logfile=/home/abhirath/Desktop/Hitler/Logs/gunicornerr.log
stderr_logfile_backups=1
stdout_logfile=/home/abhirath/Desktop/Hitler/Logs/gunicornout.log
stdout_logfile_backups=1
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
startretries=0

[program:celery]
command=bash -c "./hitler.sh celery"
directory=/home/abhirath/Desktop/Hitler
environment=PATH="/home/abhirath/Desktop/Hitler"
stderr_logfile=/home/abhirath/Desktop/Hitler/Logs/celeryerr.log
stderr_logfile_backups=1
stdout_logfile=/home/abhirath/Desktop/Hitler/Logs/celeryout.log
stdout_logfile_backups=1
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
startretries=0

I don't think there's much difference between the answers. I like to do it in a Pythonic way. — Amin Alaee, Dec 02 '16 at 12:56
@MohammadAmin Actually yeah. I have to type the paths the same number of times regardless. Accepting your answer again. — Abhirath Mahipal, Dec 02 '16 at 19:17

Unable to start a Python app using Supervisord

1

2

3

2 Answers2