gunicorn 19.2 fails to start with 18.0 configuration

Question

I've got a dev server running gunicorn/Django behind nginx. As part of a wider server environment update, I attempted to upgrade gunicorn from 18.0 to 19.2.1, but the service would no longer start. (The server is running Arch and therefore uses systemctl.)

The gunicorn configuration was done by someone who is no longer at our disposal, and not really knowing gunicorn that well, I was unable to fix or even locate the issue, so I reverted back to version 18.0 and it's working for now. However, I'd like to upgrade it eventually and get the configuration in a shape where it would work. I have a feeling that the current config is suboptimal or redundant, but I have no way of knowing for sure :-).

Nothing changed in the environment (or the virtualenv that gunicorn is running in), only gunicorn itself was upgraded. Systemctl produced this error on systemctl start gunicorn:

● gunicorn.service - gunicorn daemon (production)
   Loaded: loaded (/usr/lib/systemd/system/gunicorn.service; enabled)
   Active: failed (Result: resources) since Tue 2015-02-17 20:55:41 UTC; 8s ago
  Process: 2837 ExecStop=/bin/kill -s QUIT $MAINPID (code=exited, status=0/SUCCESS)
  Process: 9608 ExecReload=/bin/kill -s HUP $MAINPID (code=exited, status=0/SUCCESS)
  Process: 5353 ExecStart=/home/django/gunicorn/run.sh (code=exited, status=0/SUCCESS)
 Main PID: 24876 (code=exited, status=0/SUCCESS)

Feb 17 20:55:41 ashima systemd[1]: PID file /home/django/gunicorn/gunicorn.pid not readable (yet?) after start.
Feb 17 20:55:41 ashima systemd[1]: gunicorn.service never wrote its PID file. Failing.
Feb 17 20:55:41 ashima systemd[1]: Failed to start gunicorn daemon (production).
Feb 17 20:55:41 ashima systemd[1]: Unit gunicorn.service entered failed state.

Trying to run the gunicorn command contained in run.sh (pasted below) manually from the shell, it simply exited immediately without producing any errors, with an exit code of 0. Nothing was logged. In fact, it looks like my predecessor disabled gunicorn logging a while back after the log file grew to an alarming size, but that's a problem for another day.

Here are the contents of the relevant files:

/usr/lib/systemd/system/gunicorn.service:

[Unit]
Description=gunicorn daemon

[Service]
Type=forking
PIDFile=/home/django/gunicorn/gunicorn.pid
User=django
WorkingDirectory=/home/django/[name_withheld]/project
ExecStart=/home/django/gunicorn/run.sh
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=false

[Install]
WantedBy=multi-user.target

/home/django/gunicorn/run.sh:

#!/bin/bash

set -e

cd /home/django/[name_withheld]/project
source /home/django/.venv/bin/activate
exec gunicorn -p /home/django/gunicorn/gunicorn.pid -c /home/django/gunicorn/config.py -e HTTPS=on [name_withheld]_site.wsgi:application

/home/django/gunicorn/config.py:

bind = 'unix:/tmp/gunicorn.sock'
backlog = 2048
workers = 16
worker_class = 'egg:gunicorn#sync'
worker_connections = 1000
timeout = 30
keepalive = 2
debug = False
spew = False
daemon = True
pidfile = None
umask = 0755
user = None
group = None
tmp_upload_dir = None
raw_env = 'HTTPS=on'
errorlog = '-'
loglevel = 'info'
accesslog = None
proc_name = None

def post_fork(server, worker):
    server.log.info("Worker spawned (pid: %s)", worker.pid)

def pre_fork(server, worker):
    pass

def pre_exec(server):
    server.log.info("Forked child, re-executing.")

def when_ready(server):
    server.log.info("Server is ready. Spawning workers")

Try running the script with 'strace -f' and see if that shows you any errors (be warned, there will be a lot of output and most of it will be gibberish to the uninitiated!). You can paste it in a pastebin somewhere and let us know here. — Tim Stoop, Feb 20 '15 at 12:32
Maybe the systemd error can help you get started on the configure issues, f.e. on the archwiki: https://wiki.archlinux.org/index.php/Systemd#Investigating_systemd_errors — Dennis Nolte, Feb 20 '15 at 13:14
After `Active: failed (Result: resources)` the rest is just a successful cleanup. Check output of the systemd journal, like last 10 events with `journalctl --lines=10` — Brian, Feb 20 '15 at 14:10
Quick update: trying the suggestions presented so far will probably require a bit of downtime on the server, I'll try to arrange it ASAP. — JK Laiho, Feb 23 '15 at 07:44
When you execute /home/django/gunicorn/run.sh manually, is the service started (you should be able to see 17 instances of gunicorn running in `ps aux`). If it does not work, get rid of `debug = False`, `errorlog = '-'` and `daemon = True` lines in config, and paste error output here. After it works manually, the systemd service might need fixing too. — skarap, Feb 23 '15 at 14:36

score 1 · Accepted Answer · edited Apr 13 '17 at 12:14

(Of the comments posted to the question, I must especially call out the one by skarap, since it helped me to find a solution on my own by making gunicorn properly output errors. I wish I could award a partial bounty for this; the conversion of that comment into an answer would not yet be a complete answer, but it did help substantially.)

Turns out this was the problematic row in the config file:

worker_class = 'egg:gunicorn#sync'

It caused this error:

Error: class uri 'egg:gunicorn#sync' invalid or not found: 

[Traceback (most recent call last):
  File "/home/django/.venv/lib/python2.7/site-packages/gunicorn/util.py", line 113, in load_class
    return pkg_resources.load_entry_point(dist, section, name)
  File "/home/django/.venv/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 318, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/django/.venv/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 2220, in load_entry_point
    raise ImportError("Entry point %r not found" % ((group,name),))
ImportError: Entry point ('gunicorn.workers', 'sync') not found
]

Replacing it with worker_class = 'sync' fixed the ImportError and therefore the problem. No other config changes were necessary in the 18.0 -> 19.2.1 upgrade.

There seems to be a problem with gunicorn's documentation, which I intend to report, because at the time of writing this, the docs for v19.2.1 still state that the egg:gunicorn#[worker] syntax is valid. (The example there uses gevent, but it looks like it should apply to other types). Who knows, it may be valid in some circumstances, but in mine (gunicorn in a virtualenv, installed with pip), it's not.

gunicorn 19.2 fails to start with 18.0 configuration

1 Answers1