4

I know this question has already been asked and if you run two instances of the celery beat scheduler, your workers are bound to receive tasks twice. The thing is, I need to somehow get around this and I am kind of lost as to how to proceed. Isn't it possible to assign to each scheduler a subset of all the existing schedules, given I'm using a database to store them? This all, using celery beat, of course.

Thanks in advance for the responses!

georgi koyrushki
  • 51
  • 2
  • 4
  • Have you looked at https://stackoverflow.com/questions/7011950/distributed-celery-scheduler? The general advice is not to run two instances of the beat scheduler, as it's not meant to be run that way. – MPlanchard Jun 28 '17 at 13:52
  • 2
    Hi @MPlanchard. Yeah I have seen this one. The reason why I need multiple instances of the beat scheduler is for load balancing purposes. It is also a requirement of the project I am currently working on. The simplest solution I can think of is running celery beat on a cluster of N nodes, attaching to each a local db where the tasks the scheduler in question is responsible for are stored. Tasks to each scheduler are submitted then in a round robin fashion. I still think it must be possible for each schd. to be able to access a subset of all tasks in a shared db. It's much cleaner solution. – georgi koyrushki Jun 28 '17 at 14:39
  • @georgikoyrushki, could you please let me know the solution you arrived, I am having the same issue. – Ankur Sharma Aug 04 '18 at 08:06
  • Hey, unfortunately all instances of celery beat are exact replicas. This means that when a given task becomes due, each replica sends the same payload (a message representing the task's attributes in some way) onto the underlying message queue. Sadly, this mens that the payload will be received and acted upon multiple times (by the celery workers). The way I worked around this was to modify the celery beat code, so that each periodic task is associated with a celery beat instance. Unfortunately, explaining this is out of the scope of this thread. – georgi koyrushki Aug 05 '18 at 10:38

0 Answers0