1

How I can make a list of learners including autofselector and autotuner in benchmark and compare their performance?

I wonder how to rank learners stratified by task when we have multiple tasks

library(mlr3verse)

mod1 = AutoTuner$new(
                learner = lrn("surv.svm", type = "hybrid", diff.meth = "makediff3",
                              gamma.mu = c(0.1, 0.1)),
                resampling = rsmp("holdout"),
                measure = msr("surv.cindex"),
                terminator = trm("evals", n_evals = 10),
                tuner = tnr("random_search"))

mod2 = AutoFSelector$new(
                    learner    = as_learner(
                                 po("imputemedian", affect_columns = selector_type("numeric")) %>>%
                                 po("imputemode",   affect_columns = selector_type("factor"))  %>>% 
                                 po("scale") %>>%
                                 po("encode", method = "one-hot") %>>% 
                                 lrn("surv.coxph")) , 
                    resampling = rsmp("holdout"),
                    measure    = msr("surv.cindex"),
                    terminator = trm("evals", n_evals = 100),
                    fselector  = fs("sequential", strategy ="sbs"))

lrns = c(mod1, mod1)

design   =  benchmark_grid(tasks      = tsks(c("actg", "rats")),
                          learners    = lrns,
                          resamplings = rsmp("holdout"))

bmr = benchmark(design, store_models = TRUE, store_backends = TRUE)
Ali Alhadab
  • 101
  • 5
  • 1
    You might want to add the `r` tag, otherwise people of the R community won't see the question. – pat-s Feb 25 '22 at 07:20
  • Can you post an example with multiple tasks? Since your question is how to rank learners stratified by task when we have multiple tasks. – missuse Feb 25 '22 at 10:36
  • This is the error I get "Error: Output type of PipeOp select during training (Task) incompatible with input type of PipeOp impute median.impute mode.scale.encode.surv.coxph (Task Surv)" I have not error when I train the model outside benchmark – Ali Alhadab Mar 01 '22 at 16:11
  • I can't reproduce the error with the code you've posted. – Lars Kotthoff Mar 01 '22 at 21:27
  • @LarsKotthoff, this code used to work but there were mlr3 updates that I installed lately and caused this error to occur. I always have this problem with almost every mlr3 update where my old code does not work for some reason. Once my run is complete, I will reproduce the error with reporex – Ali Alhadab Mar 01 '22 at 22:53
  • @LarsKotthoff, as this is an identical problem occurred before with a previous mlr3 update. so I posted my reprex there https://github.com/mlr-org/mlr3/issues/757 – Ali Alhadab Mar 02 '22 at 16:48

0 Answers0