Using celery beat as a scheduler for irregular intervals?

Question

I have a single django application that allows the user to create multiple distinct blogs. Each blog needs to collect model data (e.g. number of visits, clicks, etc.) hourly/daily/weekly etc. and the interval at which data is collected may be different between blogs. Additionally, at some point in time, the users may want to change the frequency of data collection e.g. from weekly to daily on the user interface.

Looking into Periodic Tasks from the official documentation, it appears that I would have to 'hardcode' the interval values into the settings file and I can only specify the interval once e.g.

from celery.schedules import crontab

CELERYBEAT_SCHEDULE = {
    # Executes every Monday morning at 7:30 A.M
    'add-every-monday-morning': {
        'task': 'tasks.add',
        'schedule': crontab(hour=7, minute=30, day_of_week=1),
        'args': (16, 16),
    },
}

How do I go about this or is it even possible for celery to schedule multiple tasks of the same kind at different intervals AND change the values through the user interface (via AJAX)?

with `django-celery` you can use `djcelery.schedulers.DatabaseScheduler`, which allow you to manage your periodic tasks in django admin. [celery docs](http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#using-custom-scheduler-classes) — devxplorer, Nov 02 '16 at 04:46
Wow `changing the frequency of data collection` is cool. I will write the feature in https://github.com/rajasimon/beatserver — Raja Simon, Nov 02 '16 at 05:53
@devxplorer I am trying out [`django-celery-beat`](https://pypi.python.org/pypi/django_celery_beat) which appears to be replacing `django-celery` (please let me know if this is not the case). The problem I am facing right now is that when I add new PeriodicTasks, they don't execute until I restart the beat process (`celery -A proj beat -l info -S django`). — FatHippo, Nov 14 '16 at 07:50
Everything should work without restarting. But If you update periodic tasks in bulk, you will need to update the counter manually: `PeriodicTasks.changed()`, from [docs](http://django-celery-beat.readthedocs.io/en/latest/#models) — devxplorer, Nov 14 '16 at 16:12

score 0 · Answer 1 · answered Nov 02 '16 at 07:17

0

As noted by @devxplorer, django-celery provides a database backend. You could either use this to manage tasks via the Django admin, programmatically, or expose the model through an API.

from djcelery.models import PeriodicTask

PeriodicTask(
    name="My First Task",
    ...
).create()
all_tasks = PeriodicTask.objects.all()
...

Then starting the beat process with

$ celery -A proj beat -S djcelery.schedulers.DatabaseScheduler

answered Nov 02 '16 at 07:17

pnovotnak

4,341
2
27
38

I noticed that when I create periodic tasks after starting the beat process the new tasks do not run until I restart the beat process. How do I restart the beat process automatically when a new periodic task is created? – FatHippo Nov 14 '16 at 07:42
@FatHippo you could add an external file watcher--last I heard, the `--autoreload` flag was bad juju. I tried to use it once and ended up in a rabbit hole for 3 days because it broke things in a really subtle way. I think it's been removed in the latest version; http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#features-removed-for-lack-of-funding – pnovotnak Nov 14 '16 at 08:07

Using celery beat as a scheduler for irregular intervals?

1 Answers1