Why does celery return a KeyError when executing my task?

Question

I keep getting this keyError. I'm sending strings and id (integers) to the task function, so I don't think it is serialization issue. Also it says the keyerror is on the path to the function itself, not the contents. Please help.

Tasks.py:

from celery.decorators import task
from notification import models as notification

@task(ignore_result=True)
def notify_match_creation(match, home_team, away_team, home_team_captain, away_team_captain):
    notification.send(User.objects.filter(profile__teams__pk__in=(home_team, away_team)),
                      "tournaments_new_match",
                      {'match': unicode(match),
                       'home_team_captain': home_team_captain,
                       'away_team_captain': away_team_captain,
                       })

Relevant settings:

CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = "postgresql://user:pass@localhost/ahgl"
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
BROKER_VHOST = "/"

Celery output:

[Tasks]

  . apps.tournaments.tasks.notify_match_creation
  . tournaments.tasks.notify_match_creation
[2012-02-25 02:34:06,209: WARNING/MainProcess] celery@NATTOWER has started.
[2012-02-25 02:34:06,477: WARNING/PoolWorker-4] E:\Webdesign\ahgl\ENV\lib\site-packages\djcelery\loaders.py:84: UserWarn
ing: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-02-25 02:34:06,479: WARNING/PoolWorker-2] E:\Webdesign\ahgl\ENV\lib\site-packages\djcelery\loaders.py:84: UserWarn
ing: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-02-25 02:34:06,523: WARNING/PoolWorker-3] E:\Webdesign\ahgl\ENV\lib\site-packages\djcelery\loaders.py:84: UserWarn
ing: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-02-25 02:34:06,566: WARNING/PoolWorker-1] E:\Webdesign\ahgl\ENV\lib\site-packages\djcelery\loaders.py:84: UserWarn
ing: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-02-25 02:34:31,520: INFO/MainProcess] Got task from broker: apps.tournaments.tasks.notify_match_creation[4dbd6258-
5cee-49e9-8c8a-2d2105a2d52a]
[2012-02-25 02:34:31,569: ERROR/MainProcess] Task apps.tournaments.tasks.notify_match_creation[4dbd6258-5cee-49e9-8c8a-2
d2105a2d52a] raised exception: KeyError('apps.tournaments.tasks.notify_match_creation',)
Traceback (most recent call last):
  File "E:\Webdesign\ahgl\ENV\lib\site-packages\celery\concurrency\processes\pool.py", line 211, in worker
    result = (True, func(*args, **kwds))
  File "E:\Webdesign\ahgl\ENV\lib\site-packages\celery\worker\job.py", line 50, in execute_and_trace
    task = tasks[name]
KeyError: 'apps.tournaments.tasks.notify_match_creation'
[2012-02-25 02:38:29,773: WARNING/MainProcess] celeryd: Hitting Ctrl+C again will terminate all running tasks!
[2012-02-25 02:38:29,773: WARNING/MainProcess] celeryd: Warm shutdown (MainProcess)
[2012-02-25 02:38:31,779: INFO/MainProcess] process shutting down

Shankar ARUL · Answer 1 · 2022-06-04T12:14:21.887

8

In the celery output, you see that the task that gets picked up is

tournaments.tasks.notify_match_creation (#1)

and in your key error it is

KeyError: 'apps.tournaments.tasks.notify_match_creation'

before you call the celery task, make sure that it is imported with the same name (structure) as it is in the celery task that gets picked up (#1). Please refer to the link from celery docs for getting your relative imports rights.

one possible solution, when you launch celery - try

celery worker -A apps.tournaments.tasks.notify_match_creation

this could align your task names

edited Jun 04 '22 at 12:14

answered May 11 '16 at 22:24

Shankar ARUL

12,642
11
68
69

1

I must say, after struggling with this for days, your answer offered the most "clues" compared to all others I've seen. The Celery docs seem haphazard and very inconsistent so they weren't helpful. Thanks for your great insight. – nicorellius Jun 03 '22 at 14:59

score 0 · Answer 2 · answered Nov 23 '20 at 11:07

I had the exact same issue, however, the following worked for me,

in my tasks.py I changed from

from celery.task import task

@task(name="generate_and_email_membership_invoice")
def my_task_name():
      ...

to

from ..celeryconf import app # the file where you have app.autodiscover_tasks()

@app.task()
def my_task_name():
     ..

score 0 · Answer 3 · edited Aug 21 '22 at 17:29

0

Instead of using task.nameoftask it is rather a relative import as like

app.task.nameoftask

edited Aug 21 '22 at 17:29

Braiam

1
11
47
78

answered Aug 16 '22 at 00:51

Immanuel

57
1
5

score 0 · Answer 4 · answered May 10 '23 at 08:32

I got the similar KeyError below for celery worker:

Traceback (most recent call last):
  File "C:\Users\kai\AppData\Local\Programs\Python\Python39\lib\site-packages\celery\worker\consumer\consumer.py", line 591, in on_task_received
    strategy = strategies[type_]
KeyError: 'account.tasks.display'

Because I have display task in store/tasks.py as shown below:

# "store/tasks.py"

from celery import shared_task

@shared_task
def display(arg):
    return arg

Then, I used the wrong path "account.tasks.display" to use display task in store/tasks.py as shown below:

# "core/settings.py"

CELERY_BEAT_SCHEDULE = {
    "scheduled_task": {
        "task": "account.tasks.display", # Here
        "schedule": 5.0,
        "args": ["Test"],
    }
}

So to solve the error, I used the correct path "store.tasks.display" as shown below, then the error was solved:

# "core/settings.py"

CELERY_BEAT_SCHEDULE = {
    "scheduled_task": {
        "task": "store.tasks.display", # Here
        "schedule": 5.0,
        "args": ["Test"],
    }
}

Why does celery return a KeyError when executing my task?

4 Answers4