Celery beat sends the same task twice to the worker on every interval

Question

I have the following scheduled task in example_app -> tasks.py:

@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):

    sender.add_periodic_task(
        crontab(minute='*/1'),
        test.s(),
    )

@app.task
def test():
    print('test')

However this scheduled task is executed twice every minute:

celery_1       | [2022-02-08 16:53:00,044: INFO/MainProcess] Task example_app.tasks.test[a608d307-0ef8-4230-9586-830d0d900e67] received
celery_1       | [2022-02-08 16:53:00,046: INFO/MainProcess] Task example_app.tasks.test[5d5141cc-dcb5-4608-b115-295293c619a9] received
celery_1       | [2022-02-08 16:53:00,046: WARNING/ForkPoolWorker-6] test
celery_1       | [2022-02-08 16:53:00,047: WARNING/ForkPoolWorker-7] test
celery_1       | [2022-02-08 16:53:00,048: INFO/ForkPoolWorker-6] Task example_app.tasks.test[a608d307-0ef8-4230-9586-830d0d900e67] succeeded in 0.0014668999938294291s: None
celery_1       | [2022-02-08 16:53:00,048: INFO/ForkPoolWorker-7] Task example_app.tasks.test[5d5141cc-dcb5-4608-b115-295293c619a9] succeeded in 0.001373599996441044s: None

I read that this can be caused when you change django timezone from UTC, which I have done on this project. I tried this solution from another question but it doesn't stop the duplication:

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.base')


class MyCeleryApp(Celery):
    def now(self):
        """Return the current time and date as a datetime."""
        from datetime import datetime
        return datetime.now(self.timezone)


app = MyCeleryApp('tasks')

app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

settings:

CELERY_BROKER_URL = "redis://redis:6379/0"
CELERY_RESULT_BACKEND = "redis://redis:6379/0"

running in docker:


    celery:
        restart: always
        build:
          context: .
        command: celery --app=config.celery worker -l info
        depends_on:
          - db
          - redis
          - django

    celery-beat:
        restart: always
        build:
          context: .
        command: celery --app=config.celery beat -l info
        depends_on:
          - db
          - redis
          - django

have also tried docker like this:

    celery:
        restart: always
        build:
          context: .
        command: celery --app=config.celery worker -B -l info
        depends_on:
          - db
          - redis
          - django

with same result.

Not sure what is causing this.

score 0 · Answer 1 · answered Feb 08 '22 at 23:45

In case it helps anyone else, I solved the problem by removing the periodic task from tasks.py and defining it in settings. So the files look like this: tasks.py

@app.task
def test():
    print('test')

settings.py

CELERY_BEAT_SCHEDULE = {
    "test-task": {
        "task": "example_app.tasks.test",
        "args": (),
        "schedule": crontab(),
    }
}

Not sure what was wrong with my previous setup as I thought it was inline with the docs.

Celery beat sends the same task twice to the worker on every interval

1 Answers1