1

I'm using Django's cache framework with the Database backend. In my settings I have USE_TZ=True, and the TIME_ZONE of my databases is set to: America/New_York. In my dev/testing environment, I'm using SQLite and Oracle in production both of which don't support time zones, which should mean that Django will read/write date-times as America/New York time.

The problem I see is that the cache backend always determines the expiry to be in UTC - which is great - but then before the insert it simply converts this to a string without localizing and that's what gets set in the database. This means if I set the expiry as 120 seconds at 5:00 AM in America/New_York time: Django will look at the UTC time, adds the 120 seconds (so the expiry will be 9:02 AM in UTC) and it will simply insert 9:02 into the database.

However when Django reads this value back, it will be read as 9:02 in America/New_York TZ, hence there are additional 4 hours to the intended 2 mins of expiry.

Stepping through the source code I think the issue is that the logic for determining "exp" (for the insert) is different than for "current_expires" (read operation for expiry of existing record). For "exp" the value will be set as a naive timestamp with the datetime.utcfromtimestamp function and when it is adapted, it will be simply converted to a string.

Below is where timeout will be converted to a naive utc timestamp (in _base_set function of DatabaseCache):

            if timeout is None:
                exp = datetime.max
            elif settings.USE_TZ:
                exp = datetime.utcfromtimestamp(timeout)
            else:
                exp = datetime.fromtimestamp(timeout)

..then later will be 'adapted'

exp = connection.ops.adapt_datetimefield_value(exp)

However, this simply will return the UTC value as a string as the value is a naive timestamp:

    def adapt_datetimefield_value(self, value):
        if value is None:
            return None

        # Expression values are adapted by the database.
        if hasattr(value, 'resolve_expression'):
            return value

        # SQLite doesn't support tz-aware datetimes
        if timezone.is_aware(value):
            if settings.USE_TZ:
                value = timezone.make_naive(value, self.connection.timezone)
            else:
                raise ValueError("SQLite backend does not support timezone-aware datetimes when USE_TZ is False.")

        return str(value)

Whereas for "current_expires" the database's TZ is correctly considered thanks to the "convert_datatimefield_value" db converter that get's called eventually in the _base_set function:

    def convert_datetimefield_value(self, value, expression, connection):
        if value is not None:
            if not isinstance(value, datetime.datetime):
                value = parse_datetime(value)
            if settings.USE_TZ and not timezone.is_aware(value):
                value = timezone.make_aware(value, self.connection.timezone)
        return value

I think the two different approaches cause this additional 4 hours of difference in interpreting the expiry. I would expect this to be consistent for writing and read - is there something that I'm missing?

Currently, I'm using Django 2.2 and in the middle of upgrading to the 4.2 version - but I have checked, and seems to me the main logic stayed the same for this part of the source code.

lord stock
  • 1,191
  • 5
  • 32
NDani
  • 21
  • 7
  • I haven't looked into this in detail, but it certainly seems possible that this is a Django bug (especially since this combination of settings - database cache backend with databases using `TIME_ZONE` - is probably uncommon). You could check the test suite to see if this combination gets tested. In any case, if you don't get a satisfactory answer here and think it's a bug, file an [issue](https://code.djangoproject.com). – Kevin Christopher Henry May 05 '23 at 12:03
  • Thanks a lot @KevinChristopherHenry! I'll try to dig into the test suite definitely. Probably I also need to think about what is uncommon about my configuration - I certainly had the same feeling since I didn't find it any information about others running into the same issue. – NDani May 05 '23 at 16:31
  • That's discussed in the timezone and [database settings](https://docs.djangoproject.com/en/dev/ref/settings/#time-zone) documentation: "If Django manages the database and you don’t have a strong reason to do otherwise, you should leave this option unset. It’s best to store datetimes in UTC because it avoids ambiguous or nonexistent datetimes during daylight saving time changes. Also, receiving datetimes in UTC keeps datetime arithmetic simple — there’s no need to consider potential offset changes over a DST transition." – Kevin Christopher Henry May 05 '23 at 17:05
  • I know this recommendation but unfortunately in my case the second bullet point is applicable from the referenced section of the doc: my app is connecting to a legacy database and all other processes connecting to it expect datetimes to be in localtime. I wish I could change this because it is indeed painful to handle daylight saving time changes correctly.. – NDani May 05 '23 at 17:54
  • Thanks for all the pointers @KevinChristopherHenry! I looked at the current source code's test suite and I was able to confirm that this bug has already been fixed. I guess it's really high time to do the upgrade :) – NDani May 06 '23 at 16:40

1 Answers1

1

I'm posting this in case someone runs into the same issue: starting from Django 4, the above scenario should work perfectly, as expected.

This is thanks to the fix that was applied for determining "exp" which is the expiry of the 'to-be-inserted' record. From Django 4 the corresponding code fragment reads as (_base_set method of DatabaseCache):

            if timeout is None:
                exp = datetime.max
            else:
                tz = timezone.utc if settings.USE_TZ else None
                exp = datetime.fromtimestamp(timeout, tz=tz)

which means "exp" will be time aware in case USE_TZ is True, so when the adapt_datetimefield_value function is called (e.g. for sqlite):

# SQLite doesn't support tz-aware datetimes
        if timezone.is_aware(value):
            if settings.USE_TZ:
                value = timezone.make_naive(value, self.connection.timezone)
            else:
                raise ValueError(
                    "SQLite backend does not support timezone-aware datetimes when "
                    "USE_TZ is False."
                )

        return str(value)

...this will be correctly adapted to the time zone of the database (America/New_York in my case), before the insert.

NDani
  • 21
  • 7