Using pandas' groupby with shifting

Question

I am looking to use pd.rolling_mean in a groupby operation. I want to have in each group a rolling mean of the previous elemnets within the same group. Here is an example:

id    val
0     1
0     2
0     3
1     4
1     5
2     6

Grouping by id, this should be transformed into:

id    val
0     nan
0     1
0     1.5
1     nan
1     4
2     nan

Answer 1

I think you need groupby with shift and rolling, window size can be set to scalar:

df['val']=df.groupby('id')['val'].apply(lambda x: x.shift().rolling(2, min_periods=1).mean())
print (df)
   id  val
0   0  NaN
1   0  1.0
2   0  1.5
3   1  NaN
4   1  4.0
5   2  NaN

Thank you 3novak for comment - you can set window size by max length of group:

f = lambda x: x.shift().rolling(df['id'].value_counts().iloc[0], min_periods=1).mean()
df['val'] = df.groupby('id')['val'].apply(f)
print (df)
   id  val
0   0  NaN
1   0  1.0
2   0  1.5
3   1  NaN
4   1  4.0
5   2  NaN

Answer 2

I believe you want pd.Series.expanding

df.groupby('id').val.apply(lambda x: x.expanding().mean().shift())

0    NaN
1    1.0
2    1.5
3    NaN
4    4.0
5    NaN
Name: val, dtype: float64