0

I have the following code:

time = calendar.iat[i, j]
time = dt.datetime.strptime(str(time), '%Y-%m-%d %H:%M:%S%z')  # converts the time to a datetime object 
time = time.replace(tzinfo=from_zone)  # notes the current timezone of the object 
time = time.astimezone(to_zone)  # converts the object to the desired timezone 
calendar.iat[i, j] = time  # commits the change to the dataframe

where

from_zone = tz.gettz('UTC')
to_zone = tz.gettz('America/New_York')

Calendar in this case is a Pandas Dataframe that contains UTC + 0 datetimes. I need to convert these datetimes to UTC -4/-5 (see an example below as to what is happening).

before the first data point was converted

after the first data point was converted

The way I am doing this is by looping over the data-frame and replacing these one by one (not the most efficient method, but I still need to learn how to implement vectorization properly). This being said, if I execute this line of code:

calendar.iat[i, j] = time  # commits the change to the dataframe

then I get the following warning:

FutureWarning: Setitem-like behavior with mismatched timezones is deprecated and
will change in a future version. Instead of raising (or for Index, Series, and
DataFrame methods, coercing to object dtype),the value being set (or passed as a
fill_value, or inserted) will be cast to the existing DatetimeArray/DatetimeIndex
/Series/DataFrame column's timezone. To retain the old behavior, explicitly cast
to object dtype before the operation.
calendar.iat[i, j] = time  # commits the change to the dataframe

Having googled this specific alert, I did not come up with anything worthwhile. Can someone explain to me what this alert means and how would I remove it? Is Pandas suggesting here another way of changing the time zones of the dates? The code works fine in general and the dates are converted properly to their required timezone.

The imports I am using for this code are:

import pandas as pd
import datetime as dt
from datetime import time
from dateutil import tz
FObersteiner
  • 22,500
  • 8
  • 42
  • 72
Noir
  • 171
  • 1
  • 9
  • What is `from_zone` and `to_zone` ? – jezrael May 23 '22 at 06:15
  • Good question: "from zone = tz.gettz('UTC')" and "to_zone = tz.gettz('America/New_York')" where tz is "from dateutil import tz". I will update my message so this is clear. – Noir May 23 '22 at 06:19
  • 1
    in principle, if working with time zones in pandas, try using the built-in functionality [pd.to_datetime](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html), [pd.Series.dt.tz_localize](https://pandas.pydata.org/docs/reference/api/pandas.Series.tz_localize.html) and [pd.Series.dt.tz_convert](https://pandas.pydata.org/docs/reference/api/pandas.Series.dt.tz_convert.html). – FObersteiner May 23 '22 at 06:21
  • So you would say its simply the matter of the fact that my imports dont mix well and if I would switch to a full Pandas solution it should be fine? – Noir May 23 '22 at 06:24
  • Well you *can* work with native Python datetime and dateutil in combination with pandas, I'd just not recommend it (unless absolutely needed) since pure pandas in this case & my experience is less error-prone, more readable (opinionanted...) and more efficient (no loops in the Python script for example). – FObersteiner May 23 '22 at 06:29
  • maybe you could add exemplary content of `calendar` to the question. – FObersteiner May 23 '22 at 06:31
  • That makes sense, I will do so ASAP. – Noir May 23 '22 at 06:33
  • @Noir - I hope duplicate help, if not let me know. – jezrael May 23 '22 at 06:36
  • My question is why does this warning pop up @jezrael its not how to convert timzones in Pandas. I do know that there might be better methods of doing so and that mine might not be the best, but this is a unique question, as far as I can tell and I did not find an answer to it. My code is converting the timezones perfectly fine otherwise. – Noir May 23 '22 at 06:40
  • Hmm, so if use answer from dupe is not removed warning? – jezrael May 23 '22 at 06:42
  • I do not know, I will definitely try that method out, but beyond removing the error, I also want to have an explanation for the error in question. I have now also updated my post with the screenshots of the data within the calendar dataframe. – Noir May 23 '22 at 06:46
  • 1
    similar to the answers to the q that @jezrael linked, you just need to e.g. `pd.to_datetime(calendar["market_open"]).dt.tz_convert("America/New_York")` – FObersteiner May 23 '22 at 06:49
  • I will test it out and come back to you. – Noir May 23 '22 at 06:50
  • Hi @FObersteiner this is an excellent solution, much shorter than what I had. Can you perhaps also tell me if there is a way to redefine calendar["market_open"] so I don't need to declare calendar["market_open"] and calendar["market_close"] in two separate lines? Since if I leave the column name empty I get a key error. – Noir May 23 '22 at 12:18
  • not sure what you mean by "redefine" but I'd make it two separate lines anyway for readability; `calendar["market_open"] = pd.to_datetime(calendar["market_open"]).dt. ...` and the same for "market_close". You could also create new columns to keep the original info; e.g. "market_close_lt" with "lt" meaning local time for example. – FObersteiner May 23 '22 at 12:25
  • Ok fair enough, in that case I simply made two lines as you suggested: calendar["market_open"] = pd.to_datetime(calendar["market_open"]).dt.tz_convert(to_zone) calendar["market_close"] = pd.to_datetime(calendar["market_close"]).dt.tz_convert(to_zone) Thanks a lot once again! – Noir May 23 '22 at 12:29

0 Answers0