Celery worker and command line args

Question

I am refactoring my code to use celery worker.

Before I used to use argparse to pass command line args.

e.g.

if __name__ == "__main__":
     parser = argparse.ArgumentParser(description='Node')
     parser.add_argument('--environment', action="store", default='local', help="env e.g. production of development")
     environment = arg_options.environment

But now I get this error.

celery -A tasks worker --loglevel=info --environment local
celery: error: no such option: --environment

How can I add?

I don't want to use environment variable if I don't have to.

e.g export environment=development

score 10 · Answer 1 · answered Jan 29 '14 at 13:35

10

The Celery worker does not execute your __main__.

If you want to add additional command-line options you can use app.user_options, but note that it uses the optparse module, not argparse.

See this section in the docs for more:

http://docs.celeryproject.org/en/latest/userguide/extending.html#preload-options

answered Jan 29 '14 at 13:35

asksol

19,129
5
61
68

3

I understand how to get arguments into a bootstep __init__ method, but how do I then access these parameters from a task? – cacois Aug 28 '14 at 20:20
1

Or how to get them... at all frankly. hours lost on this – chuckjones242 Sep 23 '14 at 03:13

score 7 · Answer 2 · answered Sep 04 '15 at 14:59

By looking at the source code, I've figured out a way to handle this.

On celery.py add:

from celery import bootsteps
from celery.bin import Option

....
app.user_options['worker'].add(
    Option('--server', dest='api_server', default='127.0.0.1',
           help='API server to use.')
)

app.conf['API_SERVER'] = '127.0.0.1'

class ConfigBootstep(bootsteps.Step):
    def __init__(self, worker, api_server=None, **options):
        app.conf['API_SERVER'] = api_server

app.steps['worker'].add(ConfigBootstep)

Then in the file holding your tasks add this:

from celery import current_app
...
@shared_task()
def roboscope():
    API_SERVER = current_app.conf.get('API_SERVER', '127.0.0.1')

I tried making API_SERVER a module global variable by resolving it when the module is imported but it doesn't work as it's too early. As my task is quite intensive it doesn't harm to run this many times.

You can use `bootsteps.StartStopStep` class as a base instead. This way you can get the value earlier (and save it in variable or something) and even validate the value of your argument when the worker starts. — tsionyx, Jan 13 '17 at 09:25
"on celery.py add:" forgive me - but you shouldn't collide with celery's namespace. Choosing 'celery.py' will certainly break a multi-module application. — Jeff K, Dec 29 '17 at 15:06

score 0 · Answer 3 · answered Mar 20 '20 at 05:16

Can use worker option.

For example, I had to initialise mongo database in one of the task

from celery import bootsteps
from celery.bin import Option
import mongoConfig
.....


app = Celery('scraper')

app.user_options['worker'].add(
    Option('--mongo', dest='is_mongo_required', default=None, help='Mongo Required')
)

class CustomArgs(bootsteps.Step):

    def __init__(self, worker, is_mongo_required=None, **options):
        # store the api authentication
        if is_mongo_required is not None and is_mongo_required[0] == "true":
            print("Mongo initialization required")
            mongoConfig.init(required=True)


app.steps['worker'].add(CustomArgs)


app.config_from_object('celeryConfig')

Command to execute

celery worker -A scraper --concurrency=4 --loglevel=info -Q abc,bcd --mongo=true

Celery worker and command line args

3 Answers3

Linked