0

I am using the mlr3 family of packages and hyperband methods to tune machine learning models. All is going well, but I am unable to figure out how to predict the best configuration found using hyperband to new data.

Is it possible to do this directly using information stored in the resulting instance - or do I need to extract the best configuration (i.e.,instance$result) and then fit a new model using those parameters, which I can then use to predict?

In the example below (borrowed from the mlr3 book) how would one use the information stored in instance to predict to new data?

library(paradox)
library(mlr3tuning)
#> Loading required package: mlr3
library(mlr3hyperband)
library(mlr3pipelines)
set.seed(123)

# extend "classif.rpart" with "subsampling" as preprocessing step
ll = po("subsample") %>>% lrn("classif.rpart")

# extend hyperparameters of "classif.rpart" with subsampling fraction as budget
search_space = ps(
  classif.rpart.cp = p_dbl(lower = 0.001, upper = 0.1),
  classif.rpart.minsplit = p_int(lower = 1, upper = 10),
  subsample.frac = p_dbl(lower = 0.1, upper = 1, tags = "budget")
)

instance = TuningInstanceSingleCrit$new(
  task = tsk("iris"),
  learner = ll,
  resampling = rsmp("holdout"),
  measure = msr("classif.ce"),
  terminator = trm("none"), # hyperband terminates itself
  search_space = search_space
)

tuner = tnr("hyperband", eta = 3)
tuner$optimize(instance)
#> INFO  [11:33:32.366] [bbotk] Starting to optimize 3 parameter(s) with '<TunerHyperband>' and '<TerminatorNone> [list()]' 
#> INFO  [11:33:32.407] [bbotk] Amount of brackets to be evaluated = 3,  
#> INFO  [11:33:32.413] [bbotk] Start evaluation of bracket 1 
#> INFO  [11:33:32.417] [bbotk] Training 9 configs with budget of 0.111111 for each 
#> INFO  [11:33:32.418] [bbotk] Evaluating 9 configuration(s) 
#> INFO  [11:33:32.561] [mlr3]  Running benchmark with 9 resampling iterations 
#> INFO  [11:33:32.635] [mlr3]  Applying learner 'subsample.classif.rpart' on task 'iris' (iter 1/1) 
#> INFO  [11:33:32.700] [mlr3]  Applying learner 'subsample.classif.rpart' on task 'iris' (iter 1/1) 


### OUTPUT DELETED FOR CLARITY ###
 
#> INFO  [11:33:34.564] [bbotk] Finished optimizing after 22 evaluation(s) 
#> INFO  [11:33:34.565] [bbotk] Result: 
#> INFO  [11:33:34.566] [bbotk]  classif.rpart.cp classif.rpart.minsplit subsample.frac learner_param_vals 
#> INFO  [11:33:34.566] [bbotk]        0.07348139                      5      0.1111111          <list[6]> 
#> INFO  [11:33:34.566] [bbotk]   x_domain classif.ce 
#> INFO  [11:33:34.566] [bbotk]  <list[3]>       0.02
#>    classif.rpart.cp classif.rpart.minsplit subsample.frac learner_param_vals
#> 1:       0.07348139                      5      0.1111111          <list[6]>
#>     x_domain classif.ce
#> 1: <list[3]>       0.02

instance$result
#>    classif.rpart.cp classif.rpart.minsplit subsample.frac learner_param_vals
#> 1:       0.07348139                      5      0.1111111          <list[6]>
#>     x_domain classif.ce
#> 1: <list[3]>       0.02

Created on 2021-04-07 by the reprex package (v1.0.0)

MC Fitzpatrick
  • 21
  • 4
  • 1
    You can go the autotuner route: https://stackoverflow.com/questions/61622299/mlr3-predictions-to-new-data-with-parameters-from-autotune. Since you are using a graph as a learner you will need to make a graph learner - `ll <- GraphLearner$new(ll)` – missuse Apr 07 '21 at 19:09
  • 1
    or you can set the optimum hyper parameters and train a learner `ll$param_set$values <- instance$result_learner_param_vals`; `ll$train(tsk("iris"))`; `ll$predict_newdata(tsk("iris")$data(1:3))` – missuse Apr 07 '21 at 19:21

0 Answers0