1

I have a data frame in Pandas that has dates and some other data. The dates are explicitly of type datetime.date. For the example, I'm forcing that by hand. In the real problem, the frame is imported from an external source already set that way. After manipulating the indexing, I find that my dates are of class pandas._libs.tslib.Timestamp, which then causes incompatibilities with later code. What causes that change in type / class?

Minimal working example (note the differences between lines 3 and 7):

In [1]: df = pd.DataFrame({'date' : ['02/20/2015','01/15/2016','08/21/2015'],  'i' : ['Bob', 'Bob', 'Jim'] ,'v' : [1, 2, 3]})

In [2]: df['date'] = pd.to_datetime(df.date).dt.date

In [3]: print type(df.date[0])
<type 'datetime.date'>

In [4]: df.set_index(['i','date'], inplace=True)

In [5]: print type(df.loc['Bob',:].index[0])
<class 'pandas._libs.tslib.Timestamp'>

In [6]: df.reset_index(inplace=True)

In [7]: print type(df.date[0])
<class 'pandas._libs.tslib.Timestamp'>
Brick
  • 3,998
  • 8
  • 27
  • 47
  • If you plan to manipulate dates with `pandas` you *really* should work with the NumPy datetime64 and timedelta64 dtypes. Forcing `datetime` causes you to deal with an `object` dtype which complicates basically everything related to dates with pandas. – ALollz Jul 09 '19 at 18:10

1 Answers1

0

I struggled with this for hours and finally traced the issue to the multi-index. In the original context, it was hard to spot because the multi-index operation appeared in the middle of larger set of operations that included slicing, partial indexing, etc. But the bottom line is that the type of the dates is converted at line 4 in the example above when the multi-index is set, and it remains of the Pandas class after that.

If instead I just set_index('time'), e.g. a regular (not multi) index, there's no type conversion. This was also a complicating factor in tracing the cause since I simplified the indexing operations as a first step in debugging, which eliminated the issue that I was trying to trace.

Brick
  • 3,998
  • 8
  • 27
  • 47