Find gaps between pandas dateranges that overlap

Question

I'm trying to find the gaps where a set of dateranges don't overlap a larger daterange. In the below example I'd like to find each gap where daterange_1 to daterange_3 don't overlap big_daterange.

At the moment I've gotten this far, but I'm stuck in extracting the gaps from diff, if that is even the best way to go about this.

import pandas as pd

daterange_1 = pd.date_range("2009-01-01", "2010-01-01")
daterange_2 = pd.date_range("2012-01-01", "2014-01-01")
daterange_3 = pd.date_range("2016-01-01", "2019-01-01")

big_daterange = pd.date_range("2001-01-01", "2021-01-01")

combined = daterange_1.union(daterange_2).union(daterange_3)

# Get the dates in big_daterange that aren't in combined
diff = big_daterange.difference(combined)

What I'm most interested in is how to extract the gaps (where a gap is a daterange of contiguous datetimes) and create a list of them like this:

gaps = [gap_daterange_1, gap_daterange_2, ...]

Scott Boston · Accepted Answer · 2021-05-07T12:34:34.597

4

Try:

s = diff.to_series()
grp = s.diff().ne(pd.Timedelta(days=1)).cumsum()
gaps = s.groupby(grp).agg(['min','max'])
gaps

Output:

         min        max
1 2001-01-01 2008-12-31
2 2010-01-02 2011-12-31
3 2014-01-02 2015-12-31
4 2019-01-02 2021-01-01

Details:

First, convert the DateTimeIndex to a pd.Series, using to_series.
Next use diff to find the difference between two consective values. If diff is greater than one day, then mark this record as True. Use cumsum to create groups of records between True records. Lastly, groupby grp an use max and min to find the start and end of each group.

edited May 07 '21 at 12:34

answered May 06 '21 at 16:39

Scott Boston

147,308
15
139
187

This is excellent, thanks, could you add a few comments to explain what's going on? – oxidising May 07 '21 at 09:32
1

@oxidising Done. See details. – Scott Boston May 07 '21 at 12:34

Corralien · Answer 2 · 2021-05-07T10:48:47.160

You can use the reindex method.
Everywhere you find NaT, it's because the date is missing in dtiX:

dti1 = pd.date_range("2009-01-01", "2010-01-01")
dti2 = pd.date_range("2012-01-01", "2014-01-01")
dti3 = pd.date_range("2016-01-01", "2019-01-01")

dti = pd.date_range("2001-01-01", "2021-01-01")

sr = pd.concat([dti1.to_series(), dti2.to_series(), dti3.to_series()]) \
       .sort_index().reindex(dti)

sr = sr[sr.isnull()]

>>> sr.index
DatetimeIndex(['2001-01-01', '2001-01-02', '2001-01-03', '2001-01-04',
               '2001-01-05', '2001-01-06', '2001-01-07', '2001-01-08',
               '2001-01-09', '2001-01-10',
               ...
               '2020-12-23', '2020-12-24', '2020-12-25', '2020-12-26',
               '2020-12-27', '2020-12-28', '2020-12-29', '2020-12-30',
               '2020-12-31', '2021-01-01'],
              dtype='datetime64[ns]', length=5111, freq=None)

Find gaps by creating groups of non-consecutive days:

sr = sr.index.to_series()
groups = sr.sub(sr.shift()) \
           .fillna(pd.Timedelta(days=1)) \
           .ne(pd.Timedelta(days=1)) \
           .cumsum()
gaps = sr.groupby(groups).agg(['min','max'])

>>> gaps
         min        max
0 2001-01-01 2008-12-31
1 2010-01-02 2011-12-31
2 2014-01-02 2015-12-31
3 2019-01-02 2021-01-01

Can you show me how to extract the gaps from the difference between the dateranges? Otherwise this doesn't really answer my question. — oxidising, May 06 '21 at 16:25
Changing `sr1` to `sr` in your grouping code, this now outputs what I was after, thanks. — oxidising, May 07 '21 at 09:39

Find gaps between pandas dateranges that overlap

2 Answers2