Python Pandas, converting datetime data into datetime.utc. whilst ignoring the timezones

Question

I have a pandas dataframe with the following data:

    PointId      Time                  Value
0  Brabrand1_1  2020-10-02 23:58:14+02  0.9754
1  Brabrand1_1  2020-02-23 21:03:00+01  2.1414
2  Brabrand1_1  2020-02-23 21:33:00+01  2.1406
3  Brabrand1_1  2019-01-29 18:03:32+01  1.0390
4  Brabrand1_1  2019-01-29 18:33:32+01  1.0390
5  Brabrand1_1  2019-01-29 19:03:32+01  1.0390
6  Brabrand1_1  2019-01-29 19:33:32+01  1.0399
7  Brabrand1_1  2019-01-29 20:03:32+02  1.0000

I want the Time data to be UTC, ignoring the +01, +02 ... whats a good way of removing the +01 or +02... is it best to convert to string and remove or is there a more appropirate way of converting to datetime and then converting to utc? Thanks

score 1 · Answer 1 · answered Oct 12 '20 at 15:57

1

You can change the time zone without changing the time with:

import dateutil.tz

new_datetime = old_datetime.replace(tzinfo=dateutil.tz.tzutc())

For your dataframe, you can use:

df['Time'] = df['Time'].apply(lambda old_datetime: old_datetime.replace(tzinfo=dateutil.tz.tzutc()))

If you are using python 3.9, you can use zoneinfo instead of dateutil.tz. You should avoid using pytz, although it wouldn't cause any problems in this situation.

https://blog.ganssle.io/articles/2018/03/pytz-fastest-footgun.html

answered Oct 12 '20 at 15:57

egrubbs

331
1
5

Thanks for the help, out of curiocity why should I not use pytz? – Pelle Martin Hesketh Oct 13 '20 at 08:40
1

This link explains the problem with pytz. The time zone needs to know the date before the offset can be calculated correctly, but if you use pytz in the datetime constructor, it will compute the offset before it knows the date. https://blog.ganssle.io/articles/2018/03/pytz-fastest-footgun.html – egrubbs Oct 13 '20 at 15:12
Thanks thats really helpful :) – Pelle Martin Hesketh Oct 14 '20 at 17:46

Tamal Govinda das · Answer 2 · 2020-10-12T16:23:18.387

1

If you want your data to be in UTC:

pd.to_datetime(df['Time']).dt.tz_convert('UTC')

If you want to remove tzinfo:

df['Time'].dt.tz_localize(None)

So to drop tzinfo and convert to UTC:

pd.to_datetime(df['Time']).dt.tz_localize(None).dt.tz_convert('UTC')

edited Oct 12 '20 at 16:23

answered Oct 12 '20 at 16:12

Tamal Govinda das

575
5
11

score 0 · Answer 3 · answered Oct 13 '20 at 08:40

0

Thanks for the help, I used what you people said and had to dig a bit more but got the answer by doing this:

df['Time'] = pandas.to_datetime(df['Time'], utc=True)
df.Time.dt.tz_convert('UTC')

answered Oct 13 '20 at 08:40

Pelle Martin Hesketh

157
2
9

Python Pandas, converting datetime data into datetime.utc. whilst ignoring the timezones

3 Answers3