Change the celery workers' code in production without task loss

Question

I have a system that has important long-running tasks which are executed by Celery workers. Assume that we have deployed our application using k8s or docker-compose. How can I change the celery workers' code in production without losing the tasks that they are currently executing? In another word, I want an elegant automated way to execute all unfinished tasks with the new workers.

I’m using Redis 4.3.3 as the broker and my Celery version is 5.2.7.

I have added result_backend and tried following settings but Celery didn't reschedule the running tasks after I ran "docker-compose restart worker_service_name".

CELERY_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True

Answer 1

This answer should provide some information about running on Kubernetes.

In addition, I would recommend adding (doc):

CELERYD_PREFETCH_MULTIPLIER = 1

How many messages to prefetch at a time multiplied by the number of concurrent processes. The default is 4 (four messages for each process). The default setting is usually a good choice, however – if you have very long running tasks waiting in the queue and you have to start the workers, note that the first worker to start will receive four times the number of messages initially. Thus the tasks may not be fairly distributed to the workers.

To disable prefetching, set worker_prefetch_multiplier to 1. Changing that setting to 0 will allow the worker to keep consuming as many messages as it wants.