How to avoid SECRET_KEY error when starting Celery using Supervisor?

Question

I have a Django 1.62 application running on Debian 7.8 with Nginx 1.2.1 as my proxy server and Gunicorn 19.1.1 as my application server. I've installed Celery 3.1.7 and RabbitMQ 2.8.4 to handle asynchronous tasks. I'm trying to manage my various applications, particularly Celery, using Supervisor 3.0a8. The problem is that when I try to start Celery via Supervisor, I get this error:

ImproperlyConfigured: The SECRET_KEY setting must not be empty.

(I show the entire stacktrace at the bottom.)

All of my configuration files are kept in a "conf" directory that sits just below my "myproj" project directory like this:

conf
├── celeryconfig.py
├── celeryconfig.pyc
├── celery.py
├── __init__.py
├── middleware.py
├── settings
│   ├── base.py
│   ├── dev.py
│   ├── __init__.py
│   ├── prod.py
├── urls.py
├── wsgi.py

My production Django settings are kept in prod.py settings file which inherits my base.py base settings. I keep the secret key in my virtual environment's postactivate file and I read it into my production settings as shown below. I've verified via the Python interpreter that the secret key is present in my production settings.

# conf/settings/prod.py
from conf.settings.base import *
...
# get_env_variable is defined in base.py
SECRET_KEY = get_env_variable("SECRET_KEY")

This is the get_env_variable function which reads the secret key in:

# conf/settings/base.py
def get_env_variable(var_name):
    try:
        return os.environ[var_name]
    except KeyError:
        error_msg = "Set the %s environment variable" % var_name
        raise ImproperlyConfigured(error_msg)

This is my Supervisor configuration file. It's based on the sample file show in the Celery documentation:

# /etc/supervisor/conf.d/myproj.conf
[program:myproj]
command = /www/myproj/bin/start-gunicorn
user = root
stdout_logfile = /var/log/gunicorn/supervisor.log
redirect_stderr = true

[program:celery]
directory=/www/myproj
command=/home/myproj/venv/myproj/bin/celery worker --app=conf -l debug
user=nobody
numprocs=1
stdout_logfile=/var/log/celery/celery.log
stderr_logfile=/var/log/celery/celery.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
killasgroup=true
priority=998

Here's how I loaded the new Celery configuration using Supervisor:

sudo service supervisor stop
sudo service supervisor start

This is my Celery application file:

# conf/celery.py
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
from conf import celeryconfig

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'conf.settings')
app = Celery('conf')
app.config_from_object(celeryconfig)
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

This is my Celery configuration file:

# conf/celeryconfig.py
BROKER_URL = 'amqp://guest@localhost:5672//'
CELERY_RESULT_BACKEND = 'amqp'
CELERY_ACCEPT_CONTENT = ['json', ]
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_RESULT_EXPIRES = 3600
CELERY_SEND_TASK_ERROR_EMAILS = True

Per the Celery documentation, I've modified __init__.py:

# conf/__init__.py
from __future__ import absolute_import
from .celery import app as celery_app

I can start Celery by hand using the following command and it starts up just fine:

workon myproj  # Activate project's virtual environment
celery worker -A conf -l info

However, when I try to start it via Supervisor, I get the error I've described. Thinking that maybe Supervisor couldn't access the Django secret key because it's an environment variable, I tried hard-coding the key in my prod.py settings file (instead of reading it in via the get_env_variable function) but that didn't fix the problem.

I tried combining all my settings into the prod.py settings file that contained the actual secret key but that didn't help.

I also tried adding this environment parameter to the Supervisor config file as show below but this didn't fix the problem:

# /etc/supervisor/conf.d/myproj.conf
...
[program:celery]
environment=SECRET_KEY="(my secret key)"
directory=/www/myproj
command=/home/myproj/venv/myproj/bin/celery worker --app=conf -l debug
user=nobody
...

I tried setting the "user" to the name of the user that I start the website as but that didn't help either.

# /etc/supervisor/conf.d/myproj.conf
...
[program:celery]
environment=SECRET_KEY="(my secret key)"
directory=/www/myproj
command=/home/myproj/venv/myproj/bin/celery worker --app=conf -l debug
user=myproj
...

I read about a library called django-supervisor that is supposed to ease the integration between Django and Supervisor but I get the same error if I use that library.

Finally I read here on SO that if your secret key contains a "%" sign, Supervisor won't like it. I noticed that my key did contain one "%" sign so I escaped it like this "%%" but that didn't fix the problem either.

Can anyone see what I'm doing wrong? I've seen other questions here from users that were encountering this same error under different circumstances but I tried implementing the various solutions that were discussed and none of them fixed the problem.

Thanks very much for your ideas. And sorry for the long question but this problem has a lot of moving parts.

Here is the entire stacktrace:

Traceback (most recent call last):
  File "/home/myproj/venv/myproj/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/base.py", line 307, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/celery.py", line 761, in handle_argv
    return self.execute(command, argv)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/celery.py", line 693, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/base.py", line 270, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
    state_db=self.node_format(state_db, hostname), **kwargs
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 95, in __init__
    self.app.loader.init_worker()
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/loaders/base.py", line 128, in init_worker
    self.import_default_modules()
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/loaders/base.py", line 116, in import_default_modules
    signals.import_modules.send(sender=self.app)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/utils/dispatch/signal.py", line 166, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/fixups/django.py", line 69, in on_import_modules
    self.worker_fixup.validate_models()
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/kombu/utils/__init__.py", line 322, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/fixups/django.py", line 64, in worker_fixup
    self._worker_fixup = DjangoWorkerFixup(self.app)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/fixups/django.py", line 99, in __init__
    self._cache = import_module('django.core.cache')
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/core/cache/__init__.py", line 69, in <module>
    if DEFAULT_CACHE_ALIAS not in settings.CACHES:
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/conf/__init__.py", line 54, in __getattr__
    self._setup(name)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/conf/__init__.py", line 49, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/conf/__init__.py", line 151, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

in supervisor config, try setting secret key to `'aa'` or any simple string and try — Chillar Anand, Apr 09 '15 at 09:51
Thanks for the suggestion but that didn't solve the problem. I changed the key value, restarted my virtual environment, did "env | grep SECRET" to confirm the new secret key, and then restarted Celery via Supervisor and I still got the error. — Jim, Apr 09 '15 at 17:31
I had exactly the same symptoms. Upgrading supervisor to 3.1.3 solved the problem. I uninstalled the version from apt repositories and installed via pip. — abhaga, Apr 21 '15 at 17:41
Did you ever figure out how to solve this? I'm having same problem and it's driving me crazy — LBJ33, May 09 '21 at 13:28

score 1 · Answer 1 · answered Aug 12 '16 at 07:05

In your celery config you need to specify which settings you are using. currently you have prod.py that you are using for your production code so your celery.py should be

# conf/celery.py
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
from conf import celeryconfig

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'conf.settings.prod')
app = Celery('conf')
app.config_from_object(celeryconfig)
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

Therefore celery would know which settings file you want to use (if you have multiple settings in settings folder)

How to avoid SECRET_KEY error when starting Celery using Supervisor?

1 Answers1