Celery scheduled tasks problems with Timezone

Question

I'm using celery in a server where server time is now BST, and suddenly my scheduled tasks are executing one hour before! Previously, server time was Europe/London which was GMT but now due to day light saving it has become BST (GMT + 1)

I've configured celery to use the timezone like:

CELERY_TIMEZONE = 'Europe/London'

Then when calling tasks, I've also localized value for the eta parameter to 'Europe/London' like this:

from datetime import datetime
from pytz import timezone

locale_to_use = timezone('Europe/London')
current_time = locale_to_use.localize(datetime.now())

And used this current_time as value of eta parameter when calling task.

Now is there any mistake I'm making like localizing the eta parameter value? My server is in BST.

No problems were hapenning with this configuration before the day light saving timezone was in effect!

Edit:

To make things clear, I'm posting my code samples here:

@app.task(ignore_result=True)
def eta_test():

    logger.info('Executing eta_test on {0}'.format(datetime.now()))


def run_eta_test(hour, minute):

    now_time = datetime.now()
    target_time = now_time.replace(hour=hour, minute=minute, second=0)

    from settings import options
    from pytz import timezone

    local_zone = timezone('Europe/London')

    target_time = local_zone.localize(target_time)

    eta_test.apply_async(eta=target_time)

Then I call run_eta_test from a python console in the server, like this:

test_tasks.run_eta_test(17, 17)

That is, to execute the task on 17:17:00 on the same day. The server time was 17:15:45 and instead of scheduling it 2 seconds after, it executed the task immediately:

[2014-04-04 17:15:45,341: INFO/MainProcess] Received task: scheduling.test_tasks.eta_test[c28448d6-3a51-42f7-9df2-cb93385ff7c6] eta:[2014-04-04 17:17:00.095001+01:00]
[2014-04-04 17:15:46,820: INFO/Worker-3] Executing eta_test on 2014-04-04 17:15:46.820316
[2014-04-04 17:15:46,820: INFO/MainProcess] Task scheduling.test_tasks.eta_test[c28448d6-3a51-42f7-9df2-cb93385ff7c6] succeeded in 0.0008487419690936804s: None

Then I called the task again to be executed 1 hour and few seconds later by calling like:

test_tasks.run_eta_test(18, 17)

And instead of scheduling it to one hour and few seconds later, the task executed only few seconds later, that is one hour before:

[2014-04-04 17:16:27,703: INFO/MainProcess] Received task: scheduling.test_tasks.eta_test[f1a54d08-c12d-457f-bee8-04ca35b32242] eta:[2014-04-04 18:17:00.700327+01:00]
[2014-04-04 17:17:01,846: INFO/Worker-2] Executing eta_test on 2014-04-04 17:17:01.846561
[2014-04-04 17:17:01,847: INFO/MainProcess] Task scheduling.test_tasks.eta_test[f1a54d08-c12d-457f-bee8-04ca35b32242] succeeded in 0.0012819559779018164s: None

My server date is configured to BST like:

Fri Apr  4 17:29:10 BST 2014

Now, is the timezone in the server is being an issue?

Edited Again:

I was not able to solve the problem with the help of the answer and comments. So what I did is, used: CELERY_ENABLE_UTC = False and did not use any value for CELERY_ENABLE_UTC at all. Then, I used my server time without any localization. Celery seemed to schedule tasks correctly at my server time.

score 14 · Answer 1 · edited Jun 20 '20 at 09:12

14

You might find it easier to set your CELERY_TIMEZONE to be 'UTC'. Then, if you want to use Local times to schedule events, you can do the following:

london_tz = pytz.timezone('Europe/London')
london_dt = london_tz.localize(datetime.datetime(year, month, day, hour, min))
give_this_to_celery = london_dt.astimezone(pytz.UTC)

Admittedly, this is more work. Localize a datetime, then convert it and get a naive datetime back. But it should take care of most of the headaches from working with timezones.

Edit: you asked,

Can you please tell me what exactly CELERY_TIMEZONE does? And how celery use eta value to calculate countdown?

Celery allows you to defer the execution of a function call by specifying the eta parameter. eta is a datetime object that represent when you want the function to be run. CELERY_TIMEZONE specifies the timezone of the datetimes used for eta. So, if we set CELERY_TIMEZONE = 'America/New_York', all of our eta parameters will be interpreted as if they represented New York time.

Better is to set CELERY_TIMEZONE = 'UTC', and pass datetime objects that represent UTC timestamps. This avoids a lot of the problems caused by daylight savings time.

More information available in the docs

Edit,

please see clarification and corrections by asksol on how eta parameter is constructed in the comments.

edited Jun 20 '20 at 09:12

Community

1
1

answered Apr 01 '14 at 13:25

bgschiller

2,087
1
16
30

Can you please tell me what exactly `CELERY_TIMEZONE` does? And how celery use `eta` value to calculate countdown? Thanks! – Shafiul Apr 01 '14 at 18:07
Thank you so much. I'll try your solution soon. Can you please tell how celery calculates actual countdown time from the eta parameter? Or what actually is wrong with my code/setup? Thanks! I've added more in the question. – Shafiul Apr 04 '14 at 16:31
4

Hey! Most of this correct, however the worker does not use `CELERY_TIMEZONE` for eta, it uses the timezone in the message (which is a datetime converted to ISO-8601 format via `.isoformat()`) or if not set the eta depends on the value of the `CELERY_ENABLE_UTC` setting: if enabled the naive eta is assumed to be in UTC, if disabled the naive eta is assumed to be in the local timezone of the worker. This dates back to pre 3.0 versions when there was no timezone support. (python datetime objects are naive if it does not have a tz set, or "aware" otherwise) – asksol Apr 04 '14 at 18:00
4

Also as an example: `countdown=30` is the same as `eta=to_utc(datetime.utcnow() + timedelta(seconds=30)).astimezone(app.conf.CELERY_TIMEZONE)` where you can import `to_utc` from `celery.utils.timeutils`. – asksol Apr 04 '14 at 18:01
@bgschiller I tried the way you mentioned, but it did not work. By the way, I am using Celery 3.1. My tasks were still being called one hour before the `eta` value. Could this be an issue with `pytz`? Because in my Ubuntu server it says 'BST' timezone and I was not able to find 'BST' in pytz. – Shafiul Apr 08 '14 at 17:29
@asksol Just curious, if `eta` does not use `CELERY_TIMEZONE` where is this value used? I'm using Celery 3.1 and have only set value for `CELERY_TIMEZONE` and have not set any value for `CELERY_ENABLE_UTC`, then tried the setup in my code and @bgsschiller's way, but the problem was not fixed and my tasks are being called one hour earlier! – Shafiul Apr 08 '14 at 17:33
See my answer which is correct about this bug being fixed in Celery 4.2 -- you're welcome! – Matteius Jan 18 '19 at 16:32

score 3 · Answer 2 · answered Jan 18 '19 at 16:32

This was a bug in Celery 4 that I helped contribute the fix for in Celery 4.2 -- it is true that the work around was to set the project's timezone for celery to be UTC, however as of Celery 4.2 you can use whatever timezone you want again and celerybeat scheduling will work properly. More details in the original bug report/PR: https://github.com/celery/celery/pull/4324

score 0 · Answer 3 · answered Sep 21 '17 at 03:17

I find it strange that if we set the celery_time zone, It will adjust the eta time by add the local time to utc. but, the celery eta clock seems to trigger by utc time. now the utc time is the time+celery timezone which is wrong.

I guess the @bgschiller using the utc as default time zone is a way to solve this..

def celery_localtime_util(t):
    bj_tz = pytz.timezone('*')
    bj_dt = bj_tz.localize(t)
    return bj_dt.astimezone(pytz.UTC)

use this works...

Celery scheduled tasks problems with Timezone

3 Answers3

Edit: you asked,

Edit,

Linked