Python - pandas datetime column with multiple timezones

Question

I have a data frame with multiple users and timezones, like such:

cols = ['user', 'zone_name', 'utc_datetime']
data = [
    [1, 'Europe/Amsterdam', pd.to_datetime('2019-11-13 11:14:15')],
    [2, 'Europe/London', pd.to_datetime('2019-11-13 11:14:15')],
]

df = pd.DataFrame(data, columns=cols)

Based on this other post, I apply the following change to get the user local datetime:

df['local_datetime'] = df.groupby('zone_name')[
    'utc_datetime'
].transform(lambda x: x.dt.tz_localize(x.name))

Which outputs this:

    user    zone_name     utc_datetime          local_datetime
    1   Europe/Amsterdam  2019-11-13 11:14:15   2019-11-13 11:14:15+01:00
    2   Europe/London     2019-11-13 11:14:15   2019-11-13 11:14:15+00:00

However, the local_datetime column is an object and I cannot find a way to get it as datetime64[ns] and in the following format (desired output):

    user    zone_name     utc_datetime          local_datetime
    1   Europe/Amsterdam  2019-11-13 11:14:15   2019-11-13 12:14:15
    2   Europe/London     2019-11-13 11:14:15   2019-11-13 11:14:15

score 3 · Accepted Answer · edited Jan 29 '20 at 14:23

3

I think you need Series.dt.tz_convert in lambda function:

df['local_datetime'] = (pd.to_datetime(df.groupby('zone_name')['utc_datetime']
    .transform(lambda x: x.dt.tz_localize('UTC').dt.tz_convert(x.name))
    .astype(str).str[:-6]))

print(df)
   user         zone_name        utc_datetime      local_datetime
0     1  Europe/Amsterdam 2019-11-13 11:14:15 2019-11-13 12:14:15
1     2     Europe/London 2019-11-13 11:14:15 2019-11-13 11:14:15

edited Jan 29 '20 at 14:23

prp

914
1
9
24

answered Jan 29 '20 at 13:31

jezrael

822,522
95
1,334
1,252

The data type is correct now :), but it should add 1 hour to the Amsterdam timezone not substract it. – prp Jan 29 '20 at 13:52
@prp - hmmm, I am not expert for timezones, but it seems output is correct, if positive number after dates in utc are subtract values, if negative then are added values, check [this](https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.Series.dt.tz_convert.html) - last 2 samples to end of page – jezrael Jan 29 '20 at 14:08
Amsterdam is +1 hour than London, but you can fix this by changing the arguments. I'll edit your answer. Thanks! – prp Jan 29 '20 at 14:19

score 3 · Answer 2 · answered Feb 20 '20 at 15:36

Relatively shorter answer using DataFrame.apply:

df['local_datetime'] = df.apply(lambda x: x.utc_datetime.tz_localize(tz = "UTC").tz_convert(x.zone_name), axis = 1)
print(df)
   user         zone_name        utc_datetime             local_datetime
0     1  Europe/Amsterdam 2019-11-13 11:14:15  2019-11-13 12:14:15+01:00
1     2     Europe/London 2019-11-13 11:14:15  2019-11-13 11:14:15+00:00

If you want to remove the time zone information, you can localize times by passing None

df['local_datetime'] = df.apply(lambda x: x.utc_datetime.tz_localize(tz = "UTC").tz_convert(x.zone_name).tz_localize(None), axis = 1)
print(df)
   user         zone_name        utc_datetime      local_datetime
0     1  Europe/Amsterdam 2019-11-13 11:14:15 2019-11-13 12:14:15
1     2     Europe/London 2019-11-13 11:14:15 2019-11-13 11:14:15

Python - pandas datetime column with multiple timezones

2 Answers2