Django and Celery: Tasks not imported

Question

I am using Django with Celery to run two tasks in the background related to contacts/email parsing.

Structure is:

project
   /api
   /core
       tasks.py
   settings.py

settings.py file contains:

BROKER_URL = 'django://'
BROKER_BACKEND = "djkombu.transport.DatabaseTransport"

#celery
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
BROKER_VHOST = "/"

sys.path.append(os.path.dirname(os.path.basename(__file__)))
CELERY_IMPORTS = ['project.core.tasks']

import djcelery
djcelery.setup_loader()

# ....

INSTALLED_APPS = (
    #...

    'kombu.transport.django',
    'djcelery',
)

tasks.py contains:

from celery.task import Task
from celery.registry import tasks

class ParseEmails(Task):
    #...

class ImportGMailContactsFromGoogleAccount(Task):
    #...

tasks.register(ParseEmails)
tasks.register(ImportGMailContactsFromGoogleAccount)

Also, added in wsgi.py

os.environ["CELERY_LOADER"] = "django"

Now, I have this app hosted on a WebFactional server. On my localhost this runs fine, but on the WebFaction server, where the Django app is deployed on a Apache server, I get:

2013-01-23 17:25:00,067: ERROR/MainProcess] Task project.core.tasks.ImportGMailContactsFromGoogleAccount[df84e03f-9d22-44ed-a305-24c20407f87c] raised exception: Task of kind 'project.core.tasks.ImportGMailContactsFromGoogleAccount' is not registered, please make sure it's imported.

But the tasks show up as registered. If I run

python2.7 manage.py celeryd -l info

I obtain:

 -------------- celery@web303.webfaction.com v3.0.13 (Chiastic Slide)
---- **** ----- 
--- * ***  * -- [Configuration]
-- * - **** --- . broker:      django://localhost//
- ** ---------- . app:         default:0x1e55350 (djcelery.loaders.DjangoLoader)
- ** ---------- . concurrency: 8 (processes)
- ** ---------- . events:      OFF (enable -E to monitor this worker)
- ** ---------- 
- *** --- * --- [Queues]
-- ******* ---- . celery:      exchange:celery(direct) binding:celery
--- ***** ----- 

[Tasks]
  . project.core.tasks.ImportGMailContactsFromGoogleAccount
  . project.core.tasks.ParseEmails

I thought it could be a relative import error, but I assumed the changes in settings.py and wsgi.py would prevent that.

I am thinking the multiple Python version supported by WebFactional could have to do with this, however I installed all the libraries for Python 2.7 and I am also running Django for 2.7, so there should be no problem with that.

Running in localhost using celeryd -l info the Tasks also show up in the list when I start the worker but it doesn't output the error when I call the task - it runs perfectly.

Thank you

score 1 · Answer 1 · answered Feb 05 '13 at 09:02

I had the same issue in a new Ubuntu 12.04 / Apache / mod_wsgi / Django 1.5 / Celery 3.0.13 production environment. Everything works fine on my Mac Os X 10.8 laptop and my old server (which has Celery 3.0.12), but not on the new server.

It seems there is some issue in Celery: https://github.com/celery/celery/issues/1150

My initial solution was changing my Task class based task to @task decorator based, from something like this:

class CreateInstancesTask(Task):
    def run(self, pk):
        management.call_command('create_instances', verbosity=0, pk=pk)
tasks.register(CreateInstancesTask)

to something like this:

@task()
def create_instances_task(pk):
    management.call_command('create_instances', verbosity=0, pk=pk)

Now this task seems to work, but of course I've to do some further testing...

It's curious you mentioned that, what I've been doing in the meantime that made it work was precisely changing from the Task class to the function decorator too. However this only works when running Celery in some ways. Using "celery -worker" works, but for example "celery-multi start 1" doesn't, which also gives a different result from using, in the manage.py directory, "manage.py celeryd_multi start 1". I guess the only way I can really make it work is to investigate the problem deep. — , Feb 05 '13 at 15:40

Django and Celery: Tasks not imported

1 Answers1