0

I asked this question over at RStudio community and received no answer so I figured I'd give it a go here. My question pertains to what budugulo asked here Select models with lowest RMSE but I'm wondering how I can go further and use the models with the best predictive capability against the test data and apply it across the entire original hierarchical dataset to get future observations.

I understand how to forecast into the future with one individual time series, but I'm trying to forecast a hierarchical dataset that would require too much time to forecast the best models onto all of the original time series individually to forecast future observations. Is there a way to fit the best models (using lowest RMSE) onto the original time series in a hierarchical dataset to forecast future observations 3 years into the future (2020)? I tried using refit() but to no avail.

Hopefully the code below will help towards answering my question.

library(tidyverse)
library(tsibble)
library(fable)
library(fpp3)


fit <- tourism %>%
  filter(Quarter <= yearquarter("2015 Q1")) %>%
  model(
    ets = ETS(Trips),
    arima = ARIMA(Trips)
  )

fc <- fit %>%
  forecast(new_data = filter(tourism, Quarter > yearquarter("2015 Q1")))

bestrmse <- accuracy(fc, tourism) %>%
  group_by(Region, State, Purpose) %>%
  filter(RMSE == min(RMSE)) %>%
  select(.model:Region)

bestfits <- fit %>%
  pivot_longer(cols=ets:arima, names_to = ".model", values_to = "fit") %>%
  right_join(bestrmse) %>%
  mutate(.model = "best") %>%
  pivot_wider(Region:Purpose, names_from = ".model", values_from = "fit") %>%
  as_mable(key = c(Region, State, Purpose), model = best)

#Apply 'best' models from bestfits onto original non-trained/non-tested time series and 
#forecast future observations into 2020.
cap
  • 1
  • It seems fishy to me to pick different models for each Region/State/Purpose subset of the data. That's too many parameters and seems destined to overfit, fooling you into higher confidence for the future than is justified. Seems better to pick the model which does better overall. Google "p-hacking," "garden of forking paths." – Jon Spring Aug 30 '21 at 18:51
  • [hierarchical forecasting](https://otexts.com/fpp3/hierarchical.html), either via tsibble and fable or use modeltime if you need more advanced methods and refitting. – phiver Aug 31 '21 at 08:37

0 Answers0