0

I am using mlr3 and I wanted to ask if it is possible to change the resampling method of an exiting auto_tuner().

Example:

library(mlr3verse)
# Some existing auto_tuner

learner = lrn("classif.svm",
  cost  = to_tune(1e-1, 1e5),
  gamma = to_tune(1e-1, 1),
  kernel = "radial",
  type = "C-classification"
)

at = auto_tuner(
  tuner = tnr("grid_search", resolution = 5, batch_size = 5),
  learner = learner,
  resampling = rsmp("cv", folds = 3), # The resampling I would like to change
  measure = msr("classif.ce")
)

# New resampling I would like to assign to the existing auto_tuner

new_resampling = rsmp("cv", folds = 10)

Background:

I select a model based on a nested cross validation and afterwards want to train the best model for prediction. As I use a simpler resampling inside my nested cross validatio I would like to change the resampling used by the auto_tuner to avoid creating a new one.

Markus
  • 17
  • 5

1 Answers1

0

I select a model based on a nested cross validation

Maybe you should think again about your approach:

Nested resampling is a statistical procedure to estimate the predictive performance of the model trained on the full dataset, it is not a procedure to select optimal hyperparameters. Nested resampling produces many hyperparameter configurations which should not be used to construct a final model

mlr3book

Regarding your question: No, it is not possible to change the resampling later. Just construct a new one if you want to change the resampling.

be-marc
  • 1,276
  • 5
  • 5
  • Hi and thanks for you comprehensive answer. Yes, I am very much aware of the fact that nested CV is not used to select optimal HP but to estimate the generalisation error. What I meant was that I use the generalisation error to select a algorithm (if that is the better wording), i.e., SVR and then I perform the HPO on the whole dataset to get the model I use for prediction. That’s why I wanted to change the sampling on the `auto_tuner` as for the HPO on the whole dataset I will use a different resampling then I used in the inner loop of the nested CV. – Markus Apr 21 '23 at 14:13