42

I need to merge 2 pandas dataframes together on dates, but they currently have different date types. 1 is timestamp (imported from excel) and the other is datetime.date.

Any advice?

I've tried pd.to_datetime().date but this only works on a single item(e.g. df.ix[0,0]), it won't let me apply to the entire series (e.g. df['mydates']) or the dataframe.

DYZ
  • 55,249
  • 10
  • 64
  • 93
Afo B
  • 881
  • 1
  • 7
  • 9
  • 3
    pd.to_datetime(df['mydates']) – piRSquared Jan 21 '17 at 18:45
  • 2
    Hi @piRSquared. That doesn't work unfortunately. pd.todatetime(df['my_dates']) still returns pandas.tslib.Timestamp when i access 1 element in the series (e.g. if i save that off as a new DataFrame as df2, type(df2.ix[0,0]) will still return Timestamp), and i need it to be datetime.date – Afo B Jan 22 '17 at 15:44

9 Answers9

46

I got some help from a colleague.

This appears to solve the problem posted above

pd.to_datetime(df['mydates']).apply(lambda x: x.date())

Afo B
  • 881
  • 1
  • 7
  • 9
46

Much simpler than above:

df['mydates'].dt.date
rrichter
  • 9,253
  • 5
  • 23
  • 13
15

For me this works:

from datetime import datetime
df[ts] = [datetime.fromtimestamp(x) for x in df[ts]]
Saxasmu
  • 384
  • 4
  • 8
10

You have to know if the unit of the Unix timestamp is in seconds or milliseconds. Assume that it is in seconds and assume that you have the following pandas

print(df.head())

And you get:

     timestamp XETHZUSD
0  1609459200   730.85
1  1609545600   775.01
2  1609632000   979.86
3  1609718400  1042.52
4  1609804800  1103.41

You can convert the timestamp to datetime as follows:

df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')

print(df.head())

And we get:

   timestamp XETHZUSD
0 2021-01-01   730.85
1 2021-01-02   775.01
2 2021-01-03   979.86
3 2021-01-04  1042.52
4 2021-01-05  1103.41

If the Unix timestamp was in milliseconds, then you should have typed

df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
George Pipis
  • 1,452
  • 16
  • 12
6

Another question was marked as dupe pointing to this, but it didn't include this answer, which seems the most straightforward (perhaps this method did not yet exist when this question was posted/answered):

The pandas doc shows a pandas.Timestamp.to_pydatetime method to "Convert a Timestamp object to a native Python datetime object".

codingatty
  • 2,026
  • 1
  • 23
  • 32
3

Assume time column is in timestamp integer msec format

1 day = 86400000 ms

Here you go:

day_divider = 86400000

df['time'] = df['time'].values.astype(dtype='datetime64[ms]') # for msec format

df['time'] = (df['time']/day_divider).values.astype(dtype='datetime64[D]') # for day format
Alperen Sözer
  • 371
  • 3
  • 7
1

If you need the datetime.date objects... then get them through with the .date attribute of the Timestamp

pd.to_datetime(df['mydates']).date
piRSquared
  • 285,575
  • 57
  • 475
  • 624
1

I found the following to be the most effective, when I ran into a similar issue. For instance, with the dataframe df with a series of timestmaps in column ts. 

df.ts.apply(lambda x: pd.datetime.fromtimestamp(x).date())

This makes the conversion, you can leave out the .date() suffix for datetimes. Then to alter the column on the dataframe. Like so...

df.loc[:, 'ts'] = df.ts.apply(lambda x: pd.datetime.fromtimestamp(x).date())
BrotherJack
  • 319
  • 1
  • 19
1

I was trying to convert a timestamp column to date/time, here is what I came up with:

df['Timestamp'] = df['Timestamp'].apply(lambda timestamp: datetime.fromtimestamp(timestamp))
Rafael
  • 7,002
  • 5
  • 43
  • 52