How to set celeryconfig file in in django

Question

enter image description here

I have a Django project on an Ubuntu EC2 node, which I have been using to set up an asynchronous using Celery. I've been trying to follow http://michal.karzynski.pl/blog/2014/05/18/setting-up-an-asynchronous-task-queue-for-django-using-celery-redis/

I've been able to get a basic task working at the command line, using:

(env1)ubuntu@ip-172-31-22-65:~/projects/tp$ celery --app=tp.celery:app worker --loglevel=INFO

(env1)ubuntu@ip-172-31-22-65:~/projects/tp$ celery --app=tp.celery:app worker --loglevel=INFO

 -------------- celery@ip-172-31-22-65 v3.1.17 (Cipater)
---- **** -----
--- * ***  * -- Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         tp:0x7f66a89c0470
- ** ---------- .> transport:   redis://localhost:6379/0
- ** ---------- .> results:     disabled
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery

However, if I run other celery commands like below I'm getting the following:

(env1)ubuntu@ip-172-31-22-65:~/projects/tp$ celery worker                       [2015-04-03 13:17:21,553: WARNING/MainProcess] /home/ubuntu/.virtualenvs/env1/lib/python3.4/site-packages/celery/apps/worker.py:161: 


 -------------- celery@ip-172-31-22-65 v3.1.17 (Cipater)
---- **** -----
--- * ***  * -- Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         default:0x7f1653eae7b8 (.default.Loader)
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     disabled
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[2015-04-03 13:17:21,571: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.

It appears that celery thinks I'm using amqp as a broker , but I'm using redis!!

Based on Celery tries to connect to the wrong broker, it seems likely that celery cannot find the configuration file and uses the default.

In the above question they recommend:

import your celery and add your broker like that : 

celery = Celery('task', broker='redis://127.0.0.1:6379')
celery.config_from_object(celeryconfig)

Where would I do this? is my celery.py file (below), the same as a celery config?

/projects/tp/tp/celery.py

from __future__ import absolute_import

import os
import django
from celery import Celery
from django.conf import settings

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tp.settings')
django.setup()

app = Celery('hello_django')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

tasks.py:

from __future__ import absolute_import
from celery import shared_task
from django.core.cache import cache


@shared_task
def tester1(param):
    return 'The test task executed with argument "%s" ' % param

tp/tp1/views

@csrf_exempt
def tester(request):

    tester1.delay('hi')

    return HttpResponse('hTML')

/etc/supervisor/conf.d/tp-celery.conf

[program:tp-celery]
command=/home/ubuntu/.virtualenvs/env1/bin/celery --app=tp.celery:app worker --loglevel=INFO
directory=/home/ubuntu/projects/tp
user=ubuntu
numprocs=1
stdout_logfile=/var/log/celery-worker-out.log
stderr_logfile=/var/log/celery-worker-err.log

autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

/var/log/celery-worker-out.log

 -------------- celery@ip-172-31-22-65 v3.1.17 (Cipater)
---- **** ----- 
--- * ***  * -- Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         tp:0x7fa33e424cf8
- ** ---------- .> transport:   redis://localhost:6379/0
- ** ---------- .> results:     disabled
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- 
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . testapp.tasks.tester1

put `celery.py` aside of `settings.py`.Also change, `app = Celery('hello_django')` to `app = Celery('tp')`, if your project name is `tp`. — itzMEonTV, Apr 02 '15 at 05:06
I changed app = Celery('tp'). What do you mean by ' put celery.py aside of settings.py'? — user1592380, Apr 03 '15 at 04:25
wherever `settings.py` there should be `celery.py`.That i mean.Then what is your project name ?.I mean `settings.py` in which folder ? — itzMEonTV, Apr 03 '15 at 05:26
I just thought of something, I need to reload celery after making code changes. Can I ask you how you do this? — user1592380, Apr 03 '15 at 12:34
if you are running celery using `celery --app` command, quit it(Ctrl+c) and restart it. — itzMEonTV, Apr 03 '15 at 12:38
First you have to setup all this without supervisor, because otherwise to find error always look into logs. — itzMEonTV, Apr 03 '15 at 12:44
if I run celery --app=tp.celery:app worker --loglevel=INFO at the command line it works (discussed above). However the rest of the celery functions don't work. I've added more output above so you can see. — user1592380, Apr 03 '15 at 13:24
Dont run `celery woker` only..run like`celery -A tp worker -l info`.if it working on command line then what isrest of celery function? — itzMEonTV, Apr 03 '15 at 14:59
I've modified the question to show the output, above. by 'the rest of celery function' do you mean tasks.py? I've added this above — user1592380, Apr 03 '15 at 18:26
`celery --app=tp.celery:app worker --loglevel=INFO` is working i think, then why dont you proceed to `call` tasks ? — itzMEonTV, Apr 03 '15 at 18:32
you have to call task in `tasks.py` na??For what purpose you using celery ? — itzMEonTV, Apr 04 '15 at 13:01
Ok, I understand what you are saying. I've added code from my view above. This does work when I run a worker from the command line but not when I try to run them as a daemon using supervisor. — user1592380, Apr 04 '15 at 13:28
try different log file for error and output. `celery-worker_out.log` and `celery-worker_error.log`.Then check eror logs — itzMEonTV, Apr 04 '15 at 19:40
I did this and restarted supervisor with sudo supervisorctl restart tp-celery — user1592380, Apr 04 '15 at 21:43
Interestingly the output is still being send to the old log. Shouldn't supervisor kill the old processes and use the new logs? sudo supervisorctl reread also doesn't change anything. — user1592380, Apr 04 '15 at 22:03
ok based on http://www.onurguzel.com/supervisord-restarting-and-reloading/ I did supervisorctl update, then started and stopped tp-celery — user1592380, Apr 04 '15 at 22:28
Now it is working i think ?what is the problem now ? check `error` log also if any error happened ? — itzMEonTV, Apr 05 '15 at 17:59
Yes, it is working, but the original problem remains. if I run celery worker [2015-04-03 13:17:21,571: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused. — user1592380, Apr 05 '15 at 19:02
Then why are you trying `celery worker` after all this working correct? i didnt understand that effort ? — itzMEonTV, Apr 05 '15 at 19:49
Its not just celery worker , all the celery inspect commands give the same error. I found this out after trying your answer on http://stackoverflow.com/questions/29319889/how-to-list-of-the-queued-items-in-celery — user1592380, Apr 06 '15 at 13:11
if you want that, `celery --app=tp.celery:app inspect active_queues` or simply `celery -A tp inspect active_queues` — itzMEonTV, Apr 06 '15 at 13:23
Another thing, no need of `--app`(its not error :)), simply `celery -A tp worker --loglevel=INFO` — itzMEonTV, Apr 06 '15 at 13:26
Ok , thank you, that works. I misunderstood the docs. Can you enter this as an answer please. — user1592380, Apr 06 '15 at 14:01

itzMEonTV · Accepted Answer · 2015-04-06T14:06:02.553

6

Dont run celery woker only..run like celery -A tp worker -l info. It will take default config.

For celery inspect

celery --app=tp.celery:app inspect active_queues

OR simply

celery -A tp inspect active_queues

edited Apr 06 '15 at 14:06

answered Apr 03 '15 at 14:51

itzMEonTV

19,851
4
39
49

How to set celeryconfig file in in django

1 Answers1

Linked