How to include lagged variables in statsmodel ols regression

Question

Is there a way to specify lagged independent variable in statsmodel ols regression? Here's a sample dataframe and ols model specification below. I'd like to include a lagged variable in model.

df = pd.DataFrame({
                   "y": [2,3,7,8,1],
                   "x": [8,6,2,1,9],
                   "v": [4,3,1,3,8]
                 })

Current model:

model = sm.ols(formula = 'y ~ x + v', data=df).fit()

Desired model:

model_lag = sm.ols(formula = 'y ~ (x-1) + v', data=df).fit()

score 2 · Accepted Answer · answered Oct 13 '20 at 08:52

I don't think you can call it on the fly in the formula. Maybe using the shift method? Do clarify if this is not what you need

import statsmodels.api as sm
df['xlag'] = df['x'].shift()
df

   y  x  v  xlag
0  2  8  4   NaN
1  3  6  3   8.0
2  7  2  1   6.0
3  8  1  3   2.0
4  1  9  8   1.0

sm.formula.ols(formula = 'y ~ xlag + v', data=df).fit()

score 0 · Answer 2 · answered Apr 21 '21 at 07:36

This already has an accepted answer, but to add my 2 cents:

It is good practice to verify the index before shifting (or your lag may not be what you think it is)
Can define a function which is reusable in many places in the formula

Some example code:

import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

df = pd.DataFrame({"y": [2, 3, 7, 8, 1], "x": [8, 6, 2, 1, 9], "v": [4, 3, 1, 3, 8]})


df.index = pd.PeriodIndex(
    year=[2000, 2000, 2000, 2000, 2001], quarter=[1, 2, 3, 4, 1], freq="Q", name="period"
)


def lag(x, n, validate=True):
    """Calculates the lag of a pandas Series

    Args:
        x (pd.Series): the data to lag
        n (int): How many periods to go back (lag length)
        validate (bool, optional): Validate the series index (monotonic increasing + no gaps + no duplicates). 
                                If specified, expect the index to be a pandas PeriodIndex
                                Defaults to True.

    Returns:
        pd.Series: pd.Series.shift(n) -- lagged series
    """

    if n == 0:
        return x

    if isinstance(x, pd.Series):
        if validate:
            assert x.index.is_monotonic_increasing, (
                "\u274c" + f"x.index is not monotonic_increasing"
            )
            assert x.index.is_unique, "\u274c" + f"x.index is not unique"
            idx_full = pd.period_range(start=x.index.min(), end=x.index.max(), freq=x.index.freq)
            assert np.all(x.index == idx_full), "\u274c" + f"Gaps found in x.index"
        return x.shift(n)

    return x.shift(n)


# Manually create lag as variable:
df["x_1"] = df["x"].shift(1)
smf.ols(formula="y ~ x_1 + v", data=df).fit().summary()


# Use the defined function in the formula:
smf.ols(formula="y ~ lag(x,1) + v", data=df).fit().summary()

# ... can use in multiple places too:
smf.ols(formula="y ~ lag(x,1) + lag(v, 2)", data=df).fit().summary()

How to include lagged variables in statsmodel ols regression

2 Answers2