Run function over dataframe with columns of differing length after dropna()

Question

I am trying to apply the following function over each column in a dataframe:

def hurst_lag(x):
    minlag = 200
    maxlag = 300
    lags = range(minlag, maxlag)
    tau = [sqrt(std(subtract(x.dropna()[lag:], x.dropna()[:-lag]))) for lag in lags]
    m = polyfit(log(lags), log(tau), 1)
    return m[0]*2

The function only works on non NA values. In my dataframe, the lengths of my columns differ after applying dropna(). e.g.

df = pd.DataFrame({
    'colA':[None, None, 1, 2], 
    'colB': [None, 2, 6, 4],
    'colC': [None, None, 2, 8],
    'colD': [None, 2.0, 3.0, 4.0],
})

Any ideas how to run the function over each column individually, excluding the NA values for that specific column? Many thanks

score 0 · Answer 1 · answered Jan 26 '22 at 17:50

0

Use apply to run it on the dataframe

df = df.apply(hurst_lag)

answered Jan 26 '22 at 17:50

stefan_aus_hannover

1,777
12
13

1

Hi resolved it using hurst_lag(df.apply(lambda x: pd.Series(x.dropna()), axis=0)). Many thanks – SGriff Jan 27 '22 at 11:04
I was thinking `lamda` a little after i posted that. Glad you were able to get it sorted. – stefan_aus_hannover Jan 27 '22 at 15:02

Run function over dataframe with columns of differing length after dropna()

1 Answers1