getmtime() vs datetime.now():

Question

This code prints a false warning once per year, in the night of the clock shift (central European summer time to central European time):

import os
import datetime

now = datetime.datetime.now()
age = now - datetime.datetime.fromtimestamp(os.path.getmtime(file_name))
if (age.seconds + age.days * 24 * 3600) < -180:
    print('WARN: file has timestap from future?: %s' % age)

How to make this code work even during this yearly one hour clock shift?

Update

I care only for the age, not the datetime.

score 7 · Accepted Answer · answered Nov 01 '19 at 14:56

7

The posted fragment can be easily improved by switching from local to UTC time. There are no summer (daylight saving) time changes in UTC. Just replace these two datetime functions now() -> utcnow() (docs) and fromtimestamp() -> utcfromtimestamp() (docs).

However, if the only expected output is a file age in seconds, we can directly use the timestamps (seconds from an "epoch") without any conversion:

import time
import os.path

...
age = time.time() - os.path.getmtime(file_name)

answered Nov 01 '19 at 14:56

VPfB

14,927
6
41
75

3

Using UTC in the first place is the universally right approach. – koks der drache Nov 01 '19 at 15:16
@konstantin why is the righ approach? I like this simple solution, since I (in this context) only care for the age (timedelta) not the datetime. – guettli Nov 05 '19 at 10:42
@guettli, i would say this is probably the best and simplest answer for your use case. The most important thing when comparing time is that you are comparing like for like, in this example it is a UTC timestamp vs a UTC timestamp so will always work. The reason your code did not originally work is because you were comparing objects which no longer had any relevance to a UTC timestamp as they are not timezone aware. If you intend to do more complex things, then my answer may be more useful as it's easier to work with datetime objects, but for a simple comparison this works. – KillerKode Nov 05 '19 at 11:38
1

@guettli I labeled it the "unversally right approach" because I spent too many hours if not days debugging systems and interfaces that only worked with some a priori assumptions about datetimes and timezones the received as an input. If e.g. your server does not run in the same timezone as the client and datetimes are passed along without explicit UTC offset and interpreted as local datetimes things might still somehow work out (e.g. when calculating deltas) but it's a pain to debug that could easily be avoided if everyone sticked to UTC only in the first place/as soon as possible. – koks der drache Nov 05 '19 at 12:35
1

@guettli Thank you for accepting my answer. I hope it was helpful, because I'm little bit afraid my short answer is not worth such a generous bounty and you overpaid me. Best regards _(Schöne Grüße nach Chemnitz)_ – VPfB Nov 07 '19 at 06:30
There is nothing wrong with a short answer, as long as it is readable and works I would say it is the best answer. Which is what this is. – KillerKode Nov 07 '19 at 08:15
@VPfB I love simple solutions. I use the datetime lib since several years. I am very happy that it exists. After reading your answer I laughed about myself, because I was blind and did not see the most simple solution: Just stick to "seconds since 1970" in this case. In all other cases I prefer the datetime lib. Thank you for your answer. – guettli Nov 07 '19 at 08:55

FObersteiner · Answer 2 · 2019-11-01T16:55:56.507

both your datetime objects are 'naive', meaning that they don't know about DST. datetime.now() returns the current time your machine runs on, and that might include DST. Same goes for datetime.fromtimestamp(os.path.getmtime()).

#1 - localizing your datetime objects could be an option; something like

from datetime import datetime
import tzlocal
now_aware = tzlocal.get_localzone().localize(datetime.now())
file_mtime = datetime.fromtimestamp(os.path.getmtime(file))
# assuming the file was created on a machine in the same timezone (!):
file_mtime_aware = now_aware.tzinfo.localize(file_mtime)
age = now_aware - file_mtime_aware

#2 - another option, using UTC conversion with datetime:

now = datetime.utcnow()
age = now - datetime.utcfromtimestamp(os.path.getmtime(file_name))
if (age.seconds + age.days * 24 * 3600) < -180:
    print(f'WARN: file has timestamp from future?: {age} s')

#3 - as VPfB points out in his answer, os.path.getmtime returns a UTC timestamp (check os module docs and time module docs). So the easiest solution could be to skip conversion to datetime in the first place and use only UTC timestamps; e.g. getting the current UTC timestamp as time.time().

Working with timezones can drive you mad... but there're some good resources out there, e.g. this medium post.

KillerKode · Answer 3 · 2019-11-05T10:21:23.950

Your problem is you are getting your time without it being timezone aware. So when the clocks change, you end comparing one timestamp that is from before the clock change and another that is after the clock change and your code does not see this.

You should instead get your datetime objects to be based on a specific timezone so you don't have issues with clocks changing, I recommend using the pytz module to help you with this. You can see a list of available timezones in this answer: Is there a list of Pytz Timezones?

Here is a simple code example of how you can do this with timezone aware objects:

import os
from datetime import datetime
import pytz


def get_datetime_now(timezone):
    """
    Returns timezone aware datetime object for right now
    """
    if timezone not in pytz.all_timezones:
        return None
    tz = pytz.timezone(timezone)
    dt = datetime.now().astimezone()
    return dt.astimezone(tz)


def timestamp_to_datetime(timestamp, timezone):
    """
    Returns a datetime object from a timestamp
    """
    if timezone not in pytz.all_timezones:
        return None
    tz = pytz.timezone(timezone)
    dt = datetime.fromtimestamp(timestamp).astimezone()
    return dt.astimezone(tz)


timezone = 'CET'

file_timestamp = os.path.getmtime(file_name)

now = get_datetime_now(timezone)
file_datetime = timestamp_to_datetime(file_timestamp, timezone)
age = now - file_datetime

if (age.seconds + age.days * 24 * 3600) < -180:
    print('WARN: file has timestap from future?: %s' % age)

Why is your solution better than `age = time.time() - os.path.getmtime(file_name)`. I am only interested in the age (time delta) not the datetime. — guettli, Nov 05 '19 at 10:40
If you are only interested in the time delta, then it is not. The reason I approached it this way is because you mentioned it is in the CET timezone and showed you were working with datetime objects, this approach can be useful if you are comparing times between two different timezones. If your timezones are the same then just comparing the timestamps should be sufficient. The only other consideration is to make sure your system time is synced up with a NTP server. — KillerKode, Nov 05 '19 at 10:53

getmtime() vs datetime.now():

Update

3 Answers3

Linked