12

Using Celery ver.3.1.23, I am trying to dynamically add a scheduled task to celery beat. I have one celery worker and one celery beat instance running.

Triggering a standard celery task y running task.delay() works ok. When I define a scheduled periodic task as a setting in configuration, celery beat runs it.

However what I need is to be able to add a task that runs at specified crontab at runtime. After adding a task to persistent scheduler, celery beat doesn't seem to detect the newly added new task. I can see that the celery-schedule file does have an entry with new task.

Code:

scheduler = PersistentScheduler(app=current_app, schedule_filename='celerybeat-schedule')
scheduler.add(name="adder",
          task="app.tasks.add",
          schedule=crontab(minute='*/1'),
          args=(1,2))
scheduler.close()

When I run:

print(scheduler.schedule)

I get:

{'celery.backend_cleanup': <Entry: celery.backend_cleanup celery.backend_cleanup() <crontab: 0 4 * * * (m/h/d/dM/MY)>,
'adder': <Entry: adder app.tasks.add(1, 2) <crontab: */1 * * * * (m/h/d/dM/MY)>}

​

Note that app.tasks.add has the @celery.task decorator.

sanyassh
  • 8,100
  • 13
  • 36
  • 70
Jakub Czaplicki
  • 1,787
  • 2
  • 28
  • 50

4 Answers4

1

Instead of trying to find a good workaround, I suggest you switch to the Celery Redbeat.

DejanLekic
  • 18,787
  • 4
  • 46
  • 77
0

You may solve your problem by enabling autoreloading.

However I'm not 100% sure it will work for your config file but it should if is in the CELERY_IMPORTS paths.

Hoverer note that this feature is experimental and to don't be used in production.

If you really want to have dynamic celerybeat scheduling you can always use another scheduler like the django-celery one to manage periodic tasks on db via a django admin.

Mauro Rocco
  • 4,980
  • 1
  • 26
  • 40
  • Thanks. AFAIK autoreloading would need to be an option in celery beat, and beat doesn't support autoreload. Yes I do really want to have dynamic scheduling, and I don't use django. Looks like I am ahead of my times, and dynamic task scheduling will be supported in Celery ver 4.0 – Jakub Czaplicki May 20 '16 at 09:16
  • 1
    Anyway a restart of your celery-beat shouldn't be that costly especially if you have a dedicated worker that does only scheduling. – Mauro Rocco May 20 '16 at 11:42
  • I can't restart celery-beat each time a client submits a request that may require adding new schedule. Besides, how would you do this programatically ? Btw. I've switched to python-crontab. – Jakub Czaplicki May 20 '16 at 12:32
  • 4
    In your use case it's clear that configuration based scheduling is not the proper approach just write your own scheduler with a db as backend or use one of the many out there like this https://github.com/kongluoxing/celerybeatredis. – Mauro Rocco May 20 '16 at 13:39
0

I'm having a similar problem and a solution I thought about is to pre-define some generic periodic tasks (every 1s, every 5mins, etc) and then have them getting, from DB, a list of function to be executed. Every time you want to add a new task you just add an entry in your DB.

ggaspar
  • 315
  • 2
  • 11
0

Celery beat stores all the periodically scheduled tasks in the model PeriodicTask . As a beat task can be scheduled in different ways including crontab, interval or solar. All these fields are a foreign key in the PeriodicTask model.

In order to dynamically add a scheduled task, just populate the relevant models in celery beat, the scheduler will detect changes. The changes are detected when either the count of tuple changes or save() function is called.

from django_celery_beat.models import PeriodicTask, CrontabSchedule

# -- Inside the function you want to add task dynamically 

schedule = CrontabSchedule.objects.create(minute='*/1')
task = PeriodicTask.objects.create(name='adder',
                                   task='apps.task.add', crontab=schedule)
task.save()
Ayush Pallav
  • 919
  • 9
  • 18