How to configure Celery Daemon with Django

Question

From what I can tell, there are two documents describing how to set up celery. There's "Running the worker as a daemon" and there's "First steps with Django".

In the Django docs, it says:

We also add the Django settings module as a configuration source for Celery. This means that you don’t have to use multiple configuration files, and instead configure Celery directly from the Django settings.

Which sounds awesome. However, from what I can tell, these are the files that are needed for a complete Celery daemonization:

• /etc/init.d/celeryd
• /etc/defaults/celery
• /my-proj/celery.py
• /my-proj/__init__.py

And possibly:

• /my-proj/settings.py

Boy that's a lot of files. I think I've got them all set up properly:

• /etc/init.d/celeryd has the default init.d script provided by celery.

• /etc/defaults/celery has almost nothing. Just a pointer to my app:

  export DJANGO_SETTINGS_MODULE='cl.settings'
  

• /my-proj/celery.py has the recommended file from the First Steps with Django:

  from __future__ import absolute_import
  
  import os
  
  from celery import Celery
  
  # set the default Django settings module for the 'celery' program.
  os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
  
  from django.conf import settings  # noqa
  
  app = Celery('proj')
  
  # Using a string here means the worker will not have to
  # pickle the object when using Windows.
  app.config_from_object('django.conf:settings')
  app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
  

• /my-proj/__init__.py has the recommended code from the First Steps with Django:

  from __future__ import absolute_import
  
  # This will make sure the app is always imported when
  # Django starts so that shared_task will use this app.
  from .celery import app as celery_app
  

And I have all the celery-related settings like the following in my settings.py file:

CELERY_BIN = '/var/www/.virtualenvs/my-env/bin/celery'
CELERYD_USER = 'www-data'
CELERYD_GROUP = 'www-data'
CELERYD_CONCURRENCY = 20
BROKER_URL = 'redis://'
BROKER_POOL_LIMIT = 30

Yet, when I start celery using sudo service celeryd start, it doesn't work. Instead, it's clear that it hasn't picked up my settings from my Django project, because it says:

Nov 05 20:51:59 pounamu celeryd[30190]: celery init v10.1.
Nov 05 20:51:59 pounamu celeryd[30190]: Using config script: /etc/default/celeryd
Nov 05 20:51:59 pounamu celeryd[30190]: No passwd entry for user 'celery'
Nov 05 20:51:59 pounamu su[30206]: No passwd entry for user 'celery'
Nov 05 20:51:59 pounamu su[30206]: FAILED su for celery by root
Nov 05 20:51:59 pounamu su[30206]: - ??? root:celery
Nov 05 20:51:59 pounamu systemd[1]: celeryd.service: control process exited, code=exited status=1

Any ideas where the bailing wire isn't working? Am I missing something major?

Answer 1

You are attempting to run celery as the system user "celery" which is the default used by the init script. You should create this user or you can override this by setting CELERYD_USER in /etc/defaults/celery.

Personally I prefer to use supervisord to manage celery.

Answer 2

You are missing the CELERY_APP setting. Set it in the configuration file /etc/defaults/celery or as a parameter for the worker command:

celery worker -A my-proj

Otherwise, Celery has no idea it should look at /my-proj/celery.py. The django environment variable does not affect what Celery loads.