Pandas not able to infer time series frequency, despite proper formatting?

Question

I have the following dataframe:

[1] df.index 
[1] DatetimeIndex(['1981-01-01', '1981-01-02', '1981-01-03', '1981-01-04',
               '1981-01-05', '1981-01-06', '1981-01-07', '1981-01-08',
               '1981-01-09', '1981-01-10',
               ...
               '1990-12-22', '1990-12-23', '1990-12-24', '1990-12-25',
               '1990-12-26', '1990-12-27', '1990-12-28', '1990-12-29',
               '1990-12-30', '1990-12-31'],
              dtype='datetime64[ns]', name='Date', length=3650, freq=None)

When I try to set the frequency to daily:

[2] df.index
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.5/dist-packages/pandas/core/arrays/datetimelike.py in _validate_frequency(cls, index, freq, **kwargs)
    892             if not np.array_equal(index.asi8, on_freq.asi8):
--> 893                 raise ValueError
    894         except ValueError as e:

ValueError: 

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-23-4e9469ef0e46> in <module>
----> 1 df.index.freq = 'D'

/usr/local/lib/python3.5/dist-packages/pandas/core/indexes/datetimelike.py in freq(self, value)
     98     def freq(self, value):
     99         # validation is handled by _data setter
--> 100         self._data.freq = value
    101 
    102     @property

/usr/local/lib/python3.5/dist-packages/pandas/core/arrays/datetimelike.py in freq(self, value)
    829         if value is not None:
    830             value = frequencies.to_offset(value)
--> 831             self._validate_frequency(self, value)
    832 
    833         self._freq = value

/usr/local/lib/python3.5/dist-packages/pandas/core/arrays/datetimelike.py in _validate_frequency(cls, index, freq, **kwargs)
    905                 "Inferred frequency {infer} from passed values "
    906                 "does not conform to passed frequency {passed}".format(
--> 907                     infer=inferred, passed=freq.freqstr
    908                 )
    909             )

ValueError: Inferred frequency None from passed values does not conform to passed frequency D

Note that when I do the same thing for a monthly data set, it works fine:

[1] df.index
[1] DatetimeIndex(['1985-01-01', '1985-02-01', '1985-03-01', '1985-04-01',
               '1985-05-01', '1985-06-01', '1985-07-01', '1985-08-01',
               '1985-09-01', '1985-10-01',
               ...
               '2017-04-01', '2017-05-01', '2017-06-01', '2017-07-01',
               '2017-08-01', '2017-09-01', '2017-10-01', '2017-11-01',
               '2017-12-01', '2018-01-01'],
              dtype='datetime64[ns]', name='DATE', length=397, freq=None)
[2] df.index.freq = 'MS'
[2] DatetimeIndex(['1985-01-01', '1985-02-01', '1985-03-01', '1985-04-01',
               '1985-05-01', '1985-06-01', '1985-07-01', '1985-08-01',
               '1985-09-01', '1985-10-01',
               ...
               '2017-04-01', '2017-05-01', '2017-06-01', '2017-07-01',
               '2017-08-01', '2017-09-01', '2017-10-01', '2017-11-01',
               '2017-12-01', '2018-01-01'],
              dtype='datetime64[ns]', name='DATE', length=397, freq='MS')

score 2 · Accepted Answer · answered Nov 22 '19 at 03:28

You probably do not have sequential days in your index but you do have sequential months:

ix = pd.DatetimeIndex(['1981-01-01', '1981-01-02', '1981-01-03', '1981-01-04',
                       '1981-01-05', '1981-01-06', '1981-01-07', '1981-01-08'],
                        dtype='datetime64[ns]', name='Date', freq=None)

ix.freq = 'D'

works as expected because the the days are sequential and therefore your can set the freq to days; however, if we change the last date to not be sequential: '1981-01-09' you get the error because the dates are not sequential and therefore cannot be set to freq = 'D':

ix = pd.DatetimeIndex(['1981-01-01', '1981-01-02', '1981-01-03', '1981-01-04',
                       '1981-01-05', '1981-01-06', '1981-01-07', '1981-01-09'],
                        dtype='datetime64[ns]', name='Date', freq=None)

ix.freq = 'D'

Pandas not able to infer time series frequency, despite proper formatting?

1 Answers1

Linked