0

I'm trying to configure Celery using an external file, to separate config from code. Both Celery and Flask have config.from_envvar() / config_from_envvar() methods except they behave a bit differently and what I am doing only works with Flask.

 Flask

Basically, in Flask, I do

app.config.from_object(config_class)
app.config.from_envvar('SETTINGS_FILE', silent=True)

which loads a default configuration stored in application code, then loads a settings file stored anywhere in the filesystem with a few customized settings overriding those in the default file. I just need to write that file and pass its path through an environment variable.

(More on this in Flask docs or in this answer. I find it a bit ambiguous that Flask treats the file as a Python file while the example uses settings.cfg (no .py extension) but it works fine).

Celery

When doing the same with Celery, I get this error:

ImportError: No module named '/absolute/path/to/settings'

I named the file settings.py (.py extension, in case it matters).

I don't know where to put that file. The examples I've seen, for instance in answers to this question, put the config file in the code, which is precisely what I would like to avoid.

Community
  • 1
  • 1
Jérôme
  • 13,328
  • 7
  • 56
  • 106
  • you need to make sure the module specified in 'CELERY_CONFIG_MODULE' env variable is importable by celery, while you don't want to put it under the code directory, you may add the directory of setting file to sys.path or PYTHONPATH. – georgexsh Oct 12 '17 at 19:07
  • I wanted to try that but this is not ideal because I'm doing this in a systemd config file and I don't know how to append/prepend to `PYTHONPATH`. `Environment="PYTHONPATH=/my/path:$PYTHONPATH`" won't work. See [this question](https://stackoverflow.com/questions/35439123/systemd-environment-directive-to-set-path). – Jérôme Oct 13 '17 at 07:27
  • you could set it safely, no need to append. – georgexsh Oct 13 '17 at 07:29
  • Yeah, I guess. But I'm not too fond of the idea of hardcoding PYTHONPATH in the that systemd config file. I may be wrong. For now, I think I'll settle with passing the few variables I need as env vars themselves and fetch them individually from the code (a bit verbose but I can live with this). Ideally, I'd like Celery to behave like Flask. – Jérôme Oct 13 '17 at 07:36

1 Answers1

1

since you neither want to manipulate PYTHONPATH nor put config file under application directory, the only option left will be load config from filepath:

def load_config_from_file(celery_app, filepath):
    conf = {}
    with open(filepath) as fp:
        exec(compile(fp.read(), filepath, 'exec'), {}, d)
    celery_app.config_from_object(conf)

BTW, to my knowledge most (if not all) celery config option names have no conflict with flask's, you could write them together into one file, let flask load it, then celery could just read config from flask:

celery_app.conf.update(flask_app.config)
georgexsh
  • 15,984
  • 2
  • 37
  • 62
  • +1. I was tempted by the Flask config approach. However, note that Flask config.from_envvar only loads uppercase values ([docs](http://flask.pocoo.org/docs/0.12/config/#configuring-from-files)) while Celery is moving to lowercase values. – Jérôme Oct 13 '17 at 20:33
  • yes, but I think celery would still read upper case config options for a very long time… – georgexsh Oct 14 '17 at 01:03
  • 1
    Name conflict shouldn't be an issue if you use `config_from_object` with the namespace argument: `celery_app.conf.update(flask_app.config, namespace='CELERY_')` described [here](https://github.com/celery/celery/commit/53b5fdf3c504ca667ffc8d606d2c6d6fa6f21cf6). – Jérôme Oct 16 '17 at 07:09
  • I didn't try this yet, but I'm going to accept this answer as those two methods seem like the most convenient. I'll update if I find relevant information to add when I do it. – Jérôme Oct 16 '17 at 07:38
  • For the record, I open a discussion about this on Celery's bugtracker: https://github.com/celery/celery/issues/4329 – Jérôme Oct 16 '17 at 07:46
  • Celery 5.0 will not support uppercase values (see [this comment](https://github.com/celery/celery/issues/4329#issuecomment-343952027)). – Jérôme Nov 13 '17 at 16:11
  • @Jérôme SAD! but they may change back, like sqlalchemy queue support, got removed then added back. – georgexsh Nov 13 '17 at 16:14