3

There is not much documentation available for the actual usage of django celery.backend_cleanup

Let's assume i have following 4 tasks scheduled with different interval

image

Checking DatabaseScheduler Logs I had found that only Task1 is executing on interval. 

[2018-12-28 11:21:08,241: INFO/MainProcess] Writing entries...
[2018-12-28 11:24:08,778: INFO/MainProcess] Writing entries...
[2018-12-28 11:27:09,315: INFO/MainProcess] Writing entries...
[2018-12-28 11:28:32,948: INFO/MainProcess] Scheduler: Scheduler: Sending due TASK1(project_monitor_tasks)
[2018-12-28 11:30:13,215: INFO/MainProcess] Writing entries...
[2018-12-28 11:33:13,772: INFO/MainProcess] Writing entries...
[2018-12-28 11:36:14,316: INFO/MainProcess] Writing entries...
[2018-12-28 11:39:14,868: INFO/MainProcess] Writing entries...
[2018-12-28 11:42:15,397: INFO/MainProcess] Writing entries...
[2018-12-28 11:43:55,700: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2018-12-28 11:43:55,700: INFO/MainProcess] Writing entries...
[2018-12-28 11:45:15,997: INFO/MainProcess] Writing entries...
.....
....
[2018-12-28 17:16:28,613: INFO/MainProcess] Writing entries...
[2018-12-28 17:19:29,138: INFO/MainProcess] Writing entries...
[2018-12-28 17:22:29,625: INFO/MainProcess] Writing entries...
[2018-12-28 17:25:30,140: INFO/MainProcess] Writing entries...
[2018-12-28 17:28:30,657: INFO/MainProcess] Writing entries...
[2018-12-28 17:28:32,943: INFO/MainProcess] Scheduler: Sending due TASK1(project_monitor_tasks)
[2018-12-28 17:31:33,441: INFO/MainProcess] Writing entries...
[2018-12-28 17:34:34,009: INFO/MainProcess] Writing entries...
[2018-12-28 17:37:34,578: INFO/MainProcess] Writing entries...
[2018-12-28 17:40:35,130: INFO/MainProcess] Writing entries...
[2018-12-28 17:43:35,657: INFO/MainProcess] Writing entries...
[2018-12-28 17:43:50,716: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2018-12-28 17:43:50,716: INFO/MainProcess] Writing entries...
[2018-12-28 17:46:36,266: INFO/MainProcess] Writing entries...
[2018-12-28 17:49:36,809: INFO/MainProcess] Writing entries...
[2018-12-28 17:52:37,352: INFO/MainProcess] Writing entries...

Q1) why other TASKS which are at different intervals such as 24,8,10 Hours are not executing? , I'm Assuming this is because Crontab of celery.backend_cleanup is set to every 4 Hours which is cleaning up queue tasks. Should i keep the large interval time for celery.backend_cleanup task ?

Q2) Why we should keep celery.backend_cleanup task? Does it loads new tasks on every cleanup?

Arbazz Hussain
  • 1,622
  • 2
  • 15
  • 41
  • Same Author, Same Issues. Just better answer explanation [djang-celery-beat issues #213](https://github.com/celery/django-celery-beat/issues/213) – Jay Modi Feb 06 '20 at 15:00

1 Answers1

2

Q1: We can't answer more without seeing the actual schedules, knowing your configuration of celery, or the logs past twenty-four hours. The backend_cleanup job has no effect on the broker, it's purpose is to clean up task results that have expired by deleting expired results from an RDBMS celery result backend, so it would have no effect on whether a task executes properly.

Q2: See above. You should use this task if you are using an RDBMS / database backend if you want your expired celery results to be deleted from your celery results backend database.

2ps
  • 15,099
  • 2
  • 27
  • 47