1

In my Django project, I use no SQL database at all, my primary datastore is MongoDB, through mongoengine.

I want to setup Celery to work with Redis as the broker and backend. I installed django-celery-with-redis, installed Redis (locally and in production), and tried to use the following in my settings.py:

BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = BROKER_URL
import djcelery
djcelery.setup_loader()

But when I ran the celery process and it received a task to process, when returning the result, it complained with errors that settings.DATABASES was improperly configured. That didn't make much sense since I set CELERY_RESULT_BACKEND to be Redis.

I discovered that djcelery overrides the backend on djcelery.setup_loader() to database: see DjangoLoader from the source code.

I struggled to find ways to bypass DjangoLoader's overriding, and the only way I found was by making a duplicate of djcelery/loaders.py and modifying that line:

override_backends = {
    'database': 'celery.backends.redis.RedisBackend',
}

Then in my settings.py I do:

BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = BROKER_URL
os.environ.setdefault('CELERY_LOADER',
    'myproject.utils.ugly_djcelery_hack.DjangoLoader')

Notice no djcelery.setup_loader() there any more.

This is clearly an ugly hack, is there a more elegant way of doing this?

André Staltz
  • 13,304
  • 9
  • 48
  • 58

1 Answers1

2

Can't you just set redis as CELERY_RESULT_BACKEND? See the docs for reference.

django-celery overrides default celery.backends.database for a good reason: you simply can't use SQLAlchemy with Django ORM. I see no point in routing a database backend to redis.

Krzysztof Szularz
  • 5,151
  • 24
  • 35
  • I've tried `CELERY_RESULT_BACKEND = 'redis'`, with same effects. The recommended way to set this variable is actually the whole Redis URL, see [the docs again](http://docs.celeryproject.org/en/latest/configuration.html#redis-backend-settings). Could you explain more about about your second paragraph? As far as I know SQLAlchemy is not involved here. – André Staltz Jul 17 '13 at 14:04
  • 1
    I've pointed you to docs exactly for that purpose -- not having to describe how to configure the backend. It's not only recommended way, it is the only way. Furthermore, normally `celery.backends.database` is a `SQLAlchemy` backend. All `django-celery` does is replace it with `Django ORM` based backend. Again, see the [docs](http://docs.celeryproject.org/en/latest/configuration.html#conf-database-result-backend). There is a link to `SQLAlchemy` documentation regarding connection links. – Krzysztof Szularz Jul 17 '13 at 17:36
  • Now I understand your point, and yes I agree routing `database` backend to `redis` is bad (IMO ugly hack). I think the issue here is actually that the DjangoLoader in djcelery/loaders.py doesn't yet see the settings.py variables. In [DjangoLoader.read_configuration](https://github.com/celery/django-celery/blob/master/djcelery/loaders.py#L60) it tries to get the settings attribute, and I have checked with a print statement that it is `None`. So the problem is how to make the loader get the configurations from settings.py. – André Staltz Jul 18 '13 at 09:06
  • Did you try introspecting what `settings` contain while `read_configuration` is executed? If it does contain other constants defined, then it's weird. If it doesn't contain your configuration – focus on solving that. – Krzysztof Szularz Jul 18 '13 at 09:41
  • I did debugged (`pdb`) inside `read_configuration` and it isn't loading other constants, it uses Django default values. Solving that has been tough. FYI, I use a hierarchy of settings files: common.py as a base, then development.py and production.py that both do `from settings.common import *`. This has worked fine for several months. I tried placing the CELERY_ constants in common.py or development.py, same effects. I'm starting to wonder that the Celery loader is being executed before the settings file is available. – André Staltz Jul 18 '13 at 10:16
  • Ok, seems to be some kind of circular import problem, caused by some startup functions I have in the settings file. This is not the first time I struggle with the lack of a proper space for initialization code in Django. – André Staltz Jul 18 '13 at 10:19
  • The initialization code depends on some settings. So I see two options: (1) do this ugly hack above and leave the initialization code in the settings file(s); (2) remove the ugly hack, set djcelery as I should, remove initialization code that depends on settings. Option 2 is not feasible, I really need the initialization code, so I have to stick with option 1 unless there is some other option. – André Staltz Jul 18 '13 at 12:07