python datetime dataframe add some dates if there are lack of dates than I want

Question

I have two data files and both have different periods of datetime.

As you can see below, the first 'Date' is from 2013-10-14 to 2015-11-25, and the second 'Date' is from 2014-01-01 to 2015-11-27.

If I want to make the date from 2013-10-14 to 2015-11-27 and fill the blank as np.nan, what do I have to do in the code?

If you know how to do it or any idea, please let me know.

dvv :  Date
2013-10-14   -0.038875
2013-10-15   -0.038875
2013-10-16   -0.038875
2013-10-17   -0.038875
2013-10-18   -0.038875
  
2015-11-21    0.081939
2015-11-22   -0.097986
2015-11-23   -0.096201
2015-11-24   -0.033913
2015-11-25   -0.050553
Name: dvv, Length: 773, dtype: float64
           Stations Sensor      EL      GL  Pressure   Temp     EC  Barometa
Date                                                                        
2014-01-01    JRee3    S11     NaN     NaN       NaN    NaN    NaN       NaN
2014-01-02    JRee3    S11     NaN     NaN       NaN    NaN    NaN       NaN
2014-01-02    JRee3    S11     NaN     NaN       NaN    NaN    NaN       NaN
2014-01-04    JRee3    S11     NaN     NaN       NaN    NaN    NaN       NaN
2014-01-05    JRee3    S11     NaN     NaN       NaN    NaN    NaN       NaN
            ...    ...     ...     ...       ...    ...    ...       ...
2015-11-23    JRee3    S11  213.46  202.21     99.83  14.22  105.0   1008.13
2015-11-24    JRee3    S11  213.36  202.31     99.73  14.22  105.0   1008.36
2015-11-25    JRee3    S11  213.34  202.33     99.71  14.22  105.0   1004.40
2015-11-26    JRee3    S11  213.30  202.37     99.67  14.22  105.0   1003.13
2015-11-27    JRee3    S11  213.24  202.44     99.61  14.21  105.0   1011.00
[696 rows x 8 columns]

score 1 · Answer 1 · answered Dec 19 '21 at 09:49

You can generate new dates this way (replace periods with sufficient number):

days = pd.date_range('14/10/2013', periods=365, freq='D')

You will get something like this which you can add to your dataframe:

DatetimeIndex(['2013-10-14', '2013-10-15', '2013-10-16', '2013-10-17',
               '2013-10-18', '2013-10-19', '2013-10-20', '2013-10-21',
               '2013-10-22', '2013-10-23',
               ...
               '2014-10-04', '2014-10-05', '2014-10-06', '2014-10-07',
               '2014-10-08', '2014-10-09', '2014-10-10', '2014-10-11',
               '2014-10-12', '2014-10-13'],
              dtype='datetime64[ns]', length=365, freq='D')

score 0 · Accepted Answer · answered Dec 19 '21 at 10:02

Assuming you have no missing values in the dates, then you can simply exploit pandas.date_range and an outer join.

Toy example below:

import pandas as pd
dates1 = pd.date_range('2013-10-14', '2015-11-25', freq='D')
dates2 = pd.date_range('2014-01-01', '2015-11-27', freq='D')

df1 = pd.DataFrame(data=[1]*len(dates1), index=dates1, columns=['var'])
df2 = pd.DataFrame(data=[2]*len(dates2), index=dates2, columns=['var'])

df1.merge(df2, left_index=True, right_index=True, how='outer')

python datetime dataframe add some dates if there are lack of dates than I want

2 Answers2