Celerybeat periodic tasks executed multiple times if stuck behind a long running task

Question

I am using celery 3.1.17 and redis as broker version 3.0.2 in my django program. It's running on Ubuntu 14.04.

In the settings, I defined the CELERYBEAT_SCHEDULE like:

CELERYBEAT_SCHEDULE = {
    'task1': {
        'task': 'api.tasks.exsample.task1',
        'schedule': crontab(hour=1, minute=0),
        'args': ()
    },
    'task2': {
        'task': 'api.tasks.exsample.task2',
        'schedule': crontab(hour=1, minute=30),
        'args': ()
    },
    'task3': {
        'task': 'api.tasks.exsample.task3',
        'schedule': crontab(hour=2, minute=0),
        'args': ()
    },
    'task4': {
        'task': 'api.tasks.exsample.task4',
        'schedule': crontab(hour=2, minute=30),
        'args': ()
    },
    'task5': {
        'task': 'api.tasks.exsample.task5',
        'schedule': crontab(hour=2, minute=40),
        'args': ()
    },
    'task6': {
        'task': ''api.tasks.exsample.task6',
        'schedule': crontab(hour=2, minute=50),
        'args': ()
    },
}

Here is the problem:

If the tasks finish before the schedule time of next task, it works all right. But if a task runs for a long time, say, task1 runs for two hours, then the later tasks would execute for several times each. If I restart celery and celerybeat, sometimes it still have overtime tasks, but sometimes not.

It confused me a lot. I have been reading the celery docs for a while, but could not figure out why. Could anybody tell me why this happens, how celery manages its message and task if the task queue is blocked or celery is restarted?

Task 6 has one ' to much I think. See: ''api.tasks.exsample.task6', — Richard Scholtens, Aug 10 '22 at 13:10

score 0 · Answer 1 · answered Apr 08 '16 at 04:02

In settings.py add more configuration

'''
Task hard time limit in seconds.
The worker processing the task will be killed and replaced with a new one when this is exceeded.
'''
CELERYD_TASK_TIME_LIMIT = 86400
# CELERYD_TASK_TIME_LIMIT = 10
'''
Task soft time limit in seconds.
The SoftTimeLimitExceeded exception will be raised when this is exceeded. The task can catch this to e.g.
clean up before the hard time limit comes.
'''
CELERYD_TASK_SOFT_TIME_LIMIT = 80000

Celerybeat periodic tasks executed multiple times if stuck behind a long running task

1 Answers1