I'm using facebook's Prophet library to do a forecast, but before I put it in production I want to validate it using the included cross validation package. I have a machine with 64GB RAM and Ryzen 9 5950X. I get this error: "BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending." after hours of computing.
Needless to say its frustrating, but when I look for it I can't find anyone with the same issue specifically. Many people run into this error using the "concurrent.futures" library, I assume this is built into the prophet package. The most common reason people give for this error is a lack of memory. But this is not the case for me. The memory usage never went above 20GB (so well within the limit).
I've tried on a different machine that runs a totally different platform, and it also had the same error.
This is my code:
#%%
start = time.time()
prophet = Prophet(yearly_seasonality=False,
weekly_seasonality=True,
daily_seasonality=True,
interval_width=0.9)
prophet.add_country_holidays('Netherlands')
prophet.fit(df.reset_index())
prophet_time = time.time() - start
#%%
start = time.time()
prophet_cv = cross_validation(prophet, initial='1788 days', period='24 hours', horizon = '24 hours', parallel = 'processes')
prophet_cv_time = time.time() - start
and this is my df:
df.reset_index()
ds y
0 2015-08-09 22:00:00 30.60
1 2015-08-09 23:00:00 27.19
2 2015-08-10 00:00:00 25.49
3 2015-08-10 01:00:00 24.77
4 2015-08-10 02:00:00 24.45
... ...
52651 2021-08-11 17:00:00 127.77
52652 2021-08-11 18:00:00 139.87
52653 2021-08-11 19:00:00 125.01
52654 2021-08-11 20:00:00 117.00
52655 2021-08-11 21:00:00 103.00
[52656 rows x 2 columns]
When I run my code you can see it should retrain the model 405 times and produce a 24 hour forecast every time.
INFO:fbprophet:Making 405 forecasts with cutoffs between 2020-07-02 21:00:00 and 2021-08-10 21:00:00
INFO:fbprophet:Applying in parallel with <concurrent.futures.process.ProcessPoolExecutor object at 0x00000260E14325B0>
Any help with this issue will be greatly appreciated!
Kind regards
