Celery beat serving old (removed) task

Question

Under supervisor, celery beat serves periodic tasks to celery workers for a Django app of mine. I have 4 tasks, task1, task2, task3, and task4. Recently I made a 5th task: task5.

My problem is that I commented out task5 from my workers, removed its mention from settings.py and restarted celerybeat and my celery workers. But I still see task5 periodically showing up (throwing an error in the workers' logs naturally).

Why is this happening, and how can I update the periodic tasks?

In settings.py, I have:

    import djcelery
    djcelery.setup_loader()
    # config settings for Celery Daemon

    # Redis broker
    BROKER_URL = 'redis://localhost:6379/0'

    BROKER_TRANSPORT = 'redis'

    # List of modules to import when celery starts, in myapp.tasks form. 
    CELERY_IMPORTS = ('myapp.tasks', )  

    CELERY_ALWAYS_EAGER = False

    CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
    #The backend is the resource which returns the results of a completed task from Celery. 6379 is the default port to the redis server.

    CELERY_ACCEPT_CONTENT = ['json']
    CELERY_TASK_SERIALIZER = 'json'
    CELERY_RESULT_SERIALIZER = 'json'
    CELERY_IGNORE_RESULT=True

    from datetime import timedelta

    CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

    CELERYBEAT_SCHEDULE = {
        'tasks.task1': {
            'task': 'tasks.task1',
            'schedule': timedelta(seconds=45),
        },
        'tasks.task2': {
            'task': 'tasks.task2',
            'schedule': timedelta(seconds=60),  # execute every 60 seconds
            'args': (),
        },
        'tasks.task3': {
            'task': 'tasks.task3',
            'schedule': timedelta(seconds=90),  # execute every 90 seconds
            'args': (),
        },
        'tasks.task4': {
            'task': 'tasks.task4',
            'schedule': timedelta(seconds=90),  # execute every 90 seconds
            'args': (),
        },
    }

/etc/supervisor/conf.d/celerybeat.conf contains the following:

command=python manage.py celery beat -l info
directory = /home/myuser/myproject/
environment=PATH="/home/myuser/envs/myenv/bin",VIRTUAL_ENV="/home/myuser/envs/myenv",PYTHONPATH="/home/myuser/envs/myenv/lib/python2.7:/home/myuser/envs/myenv/lib/python2.7/site-packages"
user=mhb11
numprocs=1
stdout_logfile = /etc/supervisor/logs/celerybeat.log
stderr_logfile = /etc/supervisor/logs/celerybeat.log
autostart = true
autorestart = true
startsecs=10
stopwaitsecs = 600
killasgroup=true
priority=999

Ask me for more info if you need it. Thanks in advance.

score 5 · Answer 1 · answered Jul 08 '20 at 18:58

Another possible reason is that as you're using

CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

your tasks are being stored on your database and they won't be deleted just by removing it from CELERYBEAT_SCHEDULE.

You need to delete or disable unwanted periodic tasks (easily done through djcelery app on your admin, if you have it enabled)

https://docs.celeryproject.org/en/stable/userguide/configuration.html#std:setting-beat_scheduler

score 4 · Accepted Answer · answered Sep 20 '16 at 12:00

4

I believe you already have tasks pending in the queue.

You will need to purge them. Or, if you know their id's (note that unless you overrode it, task ids are uniquely generated .. these are not the name of the task), you can also revoke them.

answered Sep 20 '16 at 12:00

creativeChips

1,177
9
12

What's the equivalent Django project command to purge/revoke tasks? – Hassan Baig Sep 20 '16 at 17:34
I am not working with django celery, so im not sure, but i did find on the web that you may find `python manage.py celery amqp queue.purge ` useful in this case... I hope you would – creativeChips Sep 20 '16 at 18:21
Thanks, will check it out. – Hassan Baig Sep 21 '16 at 02:37
@HassanBaig Just friendly reminder - if this answer was useful please upvote. If it solved your problem, please accept as right answer. Thanks! – creativeChips Sep 21 '16 at 11:27
1

I am also facing the same issue. Can someone please explain to me the reason to purge the queue. And will purging help in executing updated code? All our pending tasks will be lost too, right? – jibin mathew Jan 22 '20 at 12:43

Celery beat serving old (removed) task

2 Answers2