Celery beat: Change to individual timezone tasks causing validation error "Invalid timezone"

Question

celery --version 5.1.2 (sun-harmonics)

django --version 3.2.8

I have a celery schedule that has four tasks that run in different timezones. I am using nowfun for setting the timezones and have set CELERY_ENABLE_UTC = False in settings.py. I followed the top response on this SO post: Celery beat - different time zone per task

Note that I made this change this morning - I was running a previous version of the code without these settings.

Currently, I am saving the celery results to CELERY_RESULT_BACKEND = 'django-db'.

Since implementing the change that allows for different tasks to be run according to different timezones I am getting an error when I run celery -A backend beat -l info.

It's super long though here is the head and tail: Head:

[2021-10-29 07:29:36,059: INFO/MainProcess] beat: Starting... [2021-10-29 07:29:36,067: ERROR/MainProcess] Cannot add entry 'celery.backend_cleanup' to database schedule: ValidationError(["Invalid timezone '<LocalTimezone: UTC+00>'"]). Contents: {'task': 'celery.backend_cleanup', 'schedule': <crontab: 0 4

(m/h/d/dM/MY)>, 'options': {'expire_seconds': 43200}}

Tail:

django.core.exceptions.ValidationError: ["Invalid timezone '<LocalTimezone: UTC+00>'"]

Celery beat hangs on this last error message and I have to kill it with ctrl + c.

I went onto celery and read their instructions about manually resetting the database when timezone-related settings change - the website says:

$ python manage.py shell

from django_celery_beat.models import

PeriodicTask PeriodicTask.objects.update(last_run_at=None)

I then found some documentation that said:

Warning: If you change the Django TIME_ZONE setting your periodic task schedule will still be based on the old timezone. To fix that you would have to reset the “last run time” for each periodic task:

from django_celery_beat.models import PeriodicTask, PeriodicTasks

PeriodicTask.objects.all().update(last_run_at=None)

PeriodicTasks.changed()

Note that this will reset the state as if the periodic tasks have never run before.

So I think what's causing the problem is exactly what it says above - I changed timezones and the schedule is still running on the old UTC timezone so I need to update it, though my schedules have run before and so when I type:

>>> PeriodicTask.objects.all().update(last_run_at=None)

I get the response:

13

and then when I enter:

>>> PeriodicTasks.changed()

I get a type error:

TypeError: changed() missing 1 required positional argument: 'instance'

So my question is:

What do I do to update the PeriodTask and PeriodicTasks? What arguments should I pass to PeriodicTasks.changed() and is 13 the expected response for the first command?

Here is my celery.py:

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from django.conf import settings
from celery.schedules import crontab
import pytz
from datetime import datetime

os.environ.setdefault(
    'DJANGO_SETTINGS_MODULE',
    'backend.settings'
)

app = Celery(
    'backend'
)

app.config_from_object(
    settings,
    namespace='CELERY'
)

def uk_time():
     return datetime.now(pytz.timezone('Europe/London'))

def us_time():
    return datetime.now(pytz.timezone('EST'))

def jp_time():
    return datetime.now(pytz.timezone('Japan'))

# Celery Beat Settings
app.conf.beat_schedule={
    'generate_signals_london': {
        'task': 'signals.tasks.generate_signals',
        'schedule': crontab(
            minute=0,
            hour=8,
            nowfun=uk_time,
            day_of_week='1,2,3,4,5'
        ),
        'args': ('UK',),
    },

    'generate_signals_ny': {
        'task': 'signals.tasks.generate_signals',
        'schedule': crontab(
            minute=0,
            hour=7,
            nowfun=us_time,
            day_of_week='1,2,3,4,5'
        ),
        'args': ('NY',),
    },

    'generate_signals_nyse': {
        'task': 'signals.tasks.generate_signals',
        'schedule': crontab(
            minute=0,
            hour=9,
            nowfun=us_time,
            day_of_week='1,2,3,4,5'
        ),
        'args': ('NYSE',),
    },

    'generate_signals_asia': {
        'task': 'signals.tasks.generate_signals',
        'schedule': crontab(
            minute=0,
            hour=8,
            nowfun=jp_time,
            day_of_week='1,2,3,4,5'
        ),
        'args': ('JP',),
    },

}

app.autodiscover_tasks()

JessicaRyan · Accepted Answer · 2021-10-29T12:59:56.667

Sometimes when trying to solve a problem we are actually solving a problem created by an incorrect way of solving the initial problem.

If the answer was, "This is how you update PeriodicTask, PeriodicTasks of previously run tasks when changing timezone" then what was the original question?

The original question here was the changing of the timezone for different tasks so that those tasks incorporated DST for the various timezones. Following the solution on Celery beat - different time zone per task was not the most efficient approach to solving this problem.

To avoid needing to update the PeriodicTask and PeriodicTasks don't change CELERY_ENABLE_UTC = False and rather run everything in UTC as per the answer to this post: Celery scheduled tasks problems with Timezone

This solves the original problem.

Here is my updated celery.py and a working solution:

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from django.conf import settings
from celery.schedules import crontab
import pytz
from datetime import datetime

os.environ.setdefault(
    'DJANGO_SETTINGS_MODULE',
    'backend.settings'
)

app = Celery(
    'backend'
)

app.config_from_object(
    settings,
    namespace='CELERY'
)

dt = datetime.now()
tz = pytz.timezone('Europe/London')
dst = tz.localize(dt)

if bool(dst.dst()):
    uk_market_open = 9
else:
    uk_market_open = 8

tz = pytz.timezone('US/Eastern')
dst = tz.localize(dt)

if bool(dst.dst()):
    ny_market_open = 12
    nyse_market_open = 14
else:
    ny_market_open = 13
    nyse_market_open = 15

jp_market_open = 23

# Celery Beat Settings
app.conf.beat_schedule={
    'generate_signals_london': {
        'task': 'signals.tasks.generate_signals',
        'schedule': crontab(
            minute=0,
            hour=uk_market_open,
            day_of_week='1,2,3,4,5'
        ),
        'args': ('UK',),
    },

    'generate_signals_ny': {
        'task': 'signals.tasks.generate_signals',
        'schedule': crontab(
            minute=0,
            hour=ny_market_open,
            day_of_week='1,2,3,4,5'
        ),
        'args': ('NY',),
    },

    'generate_signals_nyse': {
        'task': 'signals.tasks.generate_signals',
        'schedule': crontab(
            minute=0,
            hour=nyse_market_open,
            day_of_week='1,2,3,4,5'
        ),
        'args': ('NYSE',),
    },

    'generate_signals_asia': {
        'task': 'signals.tasks.generate_signals',
        'schedule': crontab(
            minute=0,
            hour=jp_market_open,
            day_of_week='1,2,3,4,5'
        ),
        'args': ('JP',),
    },

}

app.autodiscover_tasks()

Celery beat: Change to individual timezone tasks causing validation error "Invalid timezone"

1 Answers1

Linked