Approach while using dynamic =True and dynamic = False in SARIMAX forecasting

Question

I have referred to previous queries in Stack Overflow but still could not come to the conclusion.

I have a dataset containing monthly commodity price. I want to predict price using SARIMAX. I want to predict price for next 24 months. Initially, I had 509 rows of actual monthly price. Now I would like to forecast price for next 24 months (or 24 rows) for which I have created new Dataframe. The new Dataframe also consists of actual Dataframe rows.

When I am using below code, I am getting this graph using "dynamic =True":

 future_df['forecast'] = results.predict(start = 508, end =533, dynamic =True)
 px.line(future_df, x='Date', y= ['Price','forecast'],template = 'plotly_dark')

When I am using below code, I am getting this graph using "dynamic =False":

future_df['forecast'] = results.predict(start = 508, end =533, dynamic =False)
px.line(future_df, x='Date', y= ['Price','forecast'],template = 'plotly_dark')

Now the actual problem comes, I am getting different graphs.

I am getting different graphs when I am using below codes using "dynamic =True" or "dynamic =False", which was not the case previously.

future_df['forecast'] = results.predict(start = 400, end =533, dynamic =True)

px.line(future_df, x='Date', y= ['Price','forecast'],template = 'plotly_dark')

future_df['forecast'] = results.predict(start = 400, end =533, dynamic =False)
px.line(future_df, x='Date', y= ['Price','forecast'],template = 'plotly_dark')

My questions

1. Why am I getting difference in graph? I can notice that dynamic =False gives better prediction in comparison to dynamic = True.
2. Which approach (dynamic =False or dynamic = True) should I follow while forecasting (start = 508, end =533) and also while validating (for example, start = 400, end =533 or start = 400, end= 508)?

I still have few more queries:

Q1) Initially I had 509 rows i.e. Monthly price for 509 time periods (= rows). Now I want to predict price for next 24 months.

I have built SARIMAX model using all 509 rows (Price). I want to validate model per graph. Which approach shall I use "dynamic = True" or "dynamic = False" ? For e.g. I want to validate price for last 133 rows price within 509 rows using plotly. I can see that "future_df['forecast'] = results.predict(start = 400, end =533, dynamic =False)" is giving me better graph in comparison to "future_df['forecast'] = results.predict(start = 400, end =533, dynamic =True)". Please advise.

Q2) My predictions using dynamic =true and false are same. Please see below code with outputs.

Forecasting using dynamic =True

future_df['forecast'] = results.predict(start = 510, end =533, dynamic =True)

## Forecasting using dynamic =False

future_df['forecast'] = results.predict(start = 510, end =533, dynamic =False)

Now I am confused which approach to use for forecasting price for next 24 months, if predictions are similar for next 24 months. Please Advise. Thanks for help in Advance!

Answer 1

When you set dynamic=True, the model continuously predicts one-step ahead (t+1) and then for the 2nd step ahead (t+2) prediction, it appends predicted value (t+1) to data, re-fits model on new expanded data then makes 2nd step ahead forecast. This is called out-of-sample prediction.

When you set dynamic=False, the model sequentially predicts one-step-ahead using the true value from previous time step instead of using predicted value. This is called in-sample prediction.

On your first comparison of plots as you predict from 509 to 533, the reason you get same plots is you are extrapolating, you do not have true values of next 24 steps that you predicted therefor regardless of setting dynamic either True or False model uses out-of-sample approach.

Since out-of-sample approach uses the last predicted value from the previous time step to predict the next value in time, as number of steps get farther, it is expected to deviate from actual values because on each step's prediction fitted model learns previous predicted step's errors as well.

Predicting from 400 to 508 with dynamic=False will have much better forecast results than dynamic=True as expected because it is in-sample approach.

Answer 2

The best thing to do is probably just plot ['Price''] on its own in a notebook or whatever and select True or False based on the real data and how it looks