0

We have our production running on AWS ECS with fargate, where we are using multiple celery workers. We have integrated flower to monitor our celery tasks with EFS as persistent db. Everything works fine unless we trigger a new deployment, once we do a new deployment, a new task will be up with new workers and flower consider them as different from the existing workers and existing workers will be considered as offline. Due to this we loose existing data after every deployment.

We tried hard coding worker names, But even after that it works the same way, the only difference is now it is not showing any offline workers after we trigger a deployment.

Please let me know your thoughts on this, is it okay to use flower to monitor celery? or is there any other tools we can use where we will not be facing this kind of issues? if flower is fine please let me know how we can fix this.

Thank you.

Sharooq CT
  • 1

1 Answers1

1

Have you tried using --persistent=True with Flower? celery flower does not show previously run tasks after restart

Here is an example of parameters to get it working (from https://github.com/mher/flower/issues/1020#issuecomment-785009471):

--persistent=True
--db=/data/flower/flower.db
--state_save_interval=5

You'll need to provide non-ephemeral storage for Flower as well. EFS should work fine.

During the deployment, I'm not sure what would happen if more than one Flower container access the same EFS. You might want to consider specifying your service deployment options with this parameters, in order to make sure there won't be overlapping containers racing for the same EFS:

  • Min running tasks: 0%
  • Max running tasks: 100%
Diogo Melo
  • 1,735
  • 3
  • 20
  • 29