Difference between first row and current row, by group

Question

I have a data set like this:

state,date,events_per_day
AM,2020-03-01,100
AM,2020-03-02,120
AM,2020-03-15,200
BA,2020-03-16,80
BA,2020-03-20,100
BA,2020-03-29,150
RS,2020-04-01,80
RS,2020-04-05,100
RS,2020-04-11,160

Now I need to compute the difference between the date in the first row of each group and the date in the current row. i.e. the first row of each group:

for group "AM" the first date is 2020-03-01;
for group "BA" the first date is 2020-03-16;
for group "RS" it is 2020-04-01.

In the end, the result I want is:

state,date,events_per_day,days_after_first_event
AM,2020-03-01,100,0
AM,2020-03-02,120,1    <--- 2020-03-02 - 2020-03-01
AM,2020-03-15,200,14   <--- 2020-03-14 - 2020-03-01
BA,2020-03-16,80,0     
BA,2020-03-20,100,4    <--- 2020-03-20 - 2020-03-16
BA,2020-03-29,150,13   <--- 2020-03-29 - 2020-03-16
RS,2020-04-01,80,0
RS,2020-04-05,100,4    <--- 2020-04-05 - 2020-04-01
RS,2020-04-11,160,10   <--- 2020-04-11 - 2020-04-01

I found How to calculate time difference by group using pandas? and it is almost to what I want. However, diff() returns the difference between consecutive lines, and I need the difference between the current line and the first line.

How can I do this?

score 6 · Answer 1 · answered Apr 30 '20 at 18:28

Option 3: groupby.transform

df['days_since_first'] = df['date'] - df.groupby('state')['date'].transform('first')

output

  state        date  events_per_day days_since_first
0    AM  2020-03-01             100           0 days
1    AM  2020-03-02             120           1 days
2    AM  2020-03-15             200          14 days
3    BA  2020-03-16              80           0 days
4    BA  2020-03-20             100           4 days
5    BA  2020-03-29             150          13 days
6    RS  2020-04-01              80           0 days
7    RS  2020-04-05             100           4 days
8    RS  2020-04-11             160          10 days

To avoid sorting, you can change trasform("first") to transform("min") — Javier Huerta, Nov 09 '22 at 14:08

score 2 · Accepted Answer · answered Apr 30 '20 at 18:25

Prepossessing:

# convert to datetime
df['date'] = pd.to_datetime(df['date'])

# extract the first dates by states:
first_dates = df.groupby('state')['date'].first()  #.min() works as well

Option 1: Index alignment

# set_index before substraction allows index alignment
df['days_since_first'] = (df.set_index('state')['date'] - first_dates).values

Option 2: map:

df['days_since_first'] = df['date'] - df['state'].map(first_dates)

Output:

  state       date  events_per_day days_since_first
0    AM 2020-03-01             100           0 days
1    AM 2020-03-02             120           1 days
2    AM 2020-03-15             200          14 days
3    BA 2020-03-16              80           0 days
4    BA 2020-03-20             100           4 days
5    BA 2020-03-29             150          13 days
6    RS 2020-04-01              80           0 days
7    RS 2020-04-05             100           4 days
8    RS 2020-04-11             160          10 days

Difference between first row and current row, by group

2 Answers2