Pandas - most recent match relative to current row

Question

I would like to add a new column to my dataframe that contains the most recent 'revenue' value where 'promotion' == 1, excluding the current row. The dataframe will always be sorted by 'day' in descending order. For rows near the bottom of the dataframe where there is not a previous row with 'promotion' == 1, it should ideally return nothing (but this is not a hard requirement as we will generally be most interested in recent records).

So if this is my dataframe:

import pandas as pd
import numpy as np
df = pd.DataFrame({'day':[10, 9, 8, 7, 6, 5, 4, 3, 2, 1],
                   'revenue':[12000, 13000, 17000, 14000, 16000,
                              15000, 19000, 17000, 12000, 11000],
                  'promotion':[0, 0, 1, 1, 0, 0, 1, 0, 1, 0]})

The new column would look like:

df['last_promo_rev'] = [17000, 17000, 14000, 19000, 19000, 19000, 12000, 12000, np.nan, np.nan]

Mark Moretto · Accepted Answer · 2020-06-23T14:06:36.027

This might work. We can use shift. This starts with your basic DataFrame example.

# Create column with valid promotion values set
df.loc[:, 'last_promo_rev'] = df.loc[df["promotion"] == 1, "revenue"]

# Shift the column by -1 to move the row up one, then backfill with revenue values.
df.loc[:, 'last_promo_rev'] = df.loc[:, "last_promo_rev"].shift(-1).bfill(axis="rows")

Output:

   day  revenue  promotion  last_promo_rev
0   10    12000          0         17000.0
1    9    13000          0         17000.0
2    8    17000          1         14000.0
3    7    14000          1         19000.0
4    6    16000          0         19000.0
5    5    15000          0         19000.0
6    4    19000          1         12000.0
7    3    17000          0         12000.0
8    2    12000          1             NaN
9    1    11000          0             NaN

this worked great when I changed `df.loc['last_promo_rev']` to `df['last_promo_rev']` in the first expression — thatguythatdoesstuff, Jun 23 '20 at 04:28
Whoops! Yep, that's how it should be. Maybe I'll do a quick edit to update the solution. — Mark Moretto, Jun 23 '20 at 14:06

score 0 · Answer 2 · answered Jun 23 '20 at 03:13

We can do it two step

s=df.revenue.mask(df.promotion!=1).bfill()
s.update(df.loc[df.promotion==1,'revenue'].shift(-1))
df['New']=s
df
   day  revenue  promotion      New
0   10    12000          0  17000.0
1    9    13000          0  17000.0
2    8    17000          1  14000.0
3    7    14000          1  19000.0
4    6    16000          0  19000.0
5    5    15000          0  19000.0
6    4    19000          1  12000.0
7    3    17000          0  12000.0
8    2    12000          1  12000.0
9    1    11000          0      NaN

Pandas - most recent match relative to current row

2 Answers2