11

I have a periodic task scheduled to run every 10 minutes. Sometimes this task completes in 2-3 minutes, sometimes it takes 20 minutes.

Is there any way using celery beats to not open the task if the previous task hasn't completed yet? I don't see an option for it in the interval settings.

cmcjake
  • 159
  • 1
  • 9

3 Answers3

8

No, Celery Beat knows nothing about the running tasks.

One way to achieve what you are trying to do is to link the task to itself. async_apply() for an example has optional parameter link and link_error which can be used to provide a signature (it can be a single task too) to run if the task finishes successfully (link) or unsuccessfully (link_error).

What I use is the following - I schedule task to run frequently (say every 5 minutes), and I use a distributed lock to make sure I always have only one instance of the task running.

Finally a reminder - you can always implement your own scheduler, and use it in your beat configuration. I was thinking about doing this in the past for exactly the same thing you want, but decided that the solution I already have is good enough for me.

DejanLekic
  • 18,787
  • 4
  • 46
  • 77
  • 2
    I ended up using the cache to indicate whether its already running or not. This does come with some bugs, for example I have a heavy nested fetching query that is designed with lots of failing points in mind. So I had to rewrite my code so that before exiting the task, always remember to update the cache variable. I do like the sound of a custom scheduler as well, and may implement this later on. – cmcjake Aug 04 '19 at 17:21
3

You can try this It provides you with a singleton base class for your tasks.

Joseph Bani
  • 644
  • 6
  • 16
1

I use Celery with Django models and I implemented a boolean has_task_running at the model level. Then with Celery signals I change the state of the flag to True when signal before_task_publish is trigged and False when a task terminates. Not simple but flexible.

Florent
  • 1,791
  • 22
  • 40