Improve speed of pandas rolling apply

Asked Nov 26 '20 at 16:33

Active Nov 26 '20 at 17:05

Viewed 837 times

I'm trying to improve the runtime speed of pandas rolling apply. My input dataframe is pretty big [df.shape=(257,2000000)] so I'm getting runtimes on the order of a number of days, which is unacceptable.

Here's my function:

def rolling_grad(y):
    model = stats.linregress(np.arange(len(y)), y)
    return model[0]

That applies over every column in the df:

grad = df.rolling(30).apply(rolling_grad)

Essentially I'm after the slope in rolling windows of size 30 for each column. I've tried using swifter and pandarallel with no luck.

Any help/advice very much appreciated. Thanks!

edited Nov 26 '20 at 17:05

asked Nov 26 '20 at 16:33

Ekbal Hussain

1

You should read this https://stackoverflow.com/questions/25583494/faster-rolling-apply-on-a-pandas-dataframe – Serge de Gosson de Varennes Nov 26 '20 at 20:28
Have you tried using Dask or pyspark? – Umar.H Dec 04 '20 at 21:52

Improve speed of pandas rolling apply

0 Answers0