How to see intermediate results from tuning in mlr in parallel?

Question

Is it possible to see results for tuning rounds when using mlr and parallelMap and parallelizing at the mlr.tuneParams level?

When I tune in serial, I see the results (hyperparameters, measures) in the Console for each combination of hyperparameters as CV finishes. Thus, if I kill a job before the tuneParams result saves, I still have some results.

When I tune in parallel, I do not know how to see intermediate results in the result that a job is terminated. Is it possible to create a log file that shows results?

Thank you!

score 3 · Accepted Answer · answered Dec 26 '19 at 12:33

This is not possible with parallelMap. In the background, mclapply() (multicore) or clusterMap() (socket) are called which do not allow progress output of workers.

You might want to try out mlr3 which relies on the future package for parallelization. With this you can select different parallel backends which might help to achieve what you want.

library("mlr")
#> Loading required package: ParamHelpers
library("parallelMap")

discrete_ps <- makeParamSet(
  makeDiscreteParam("C", values = c(0.5, 1.0, 1.5, 2.0)),
  makeDiscreteParam("sigma", values = c(0.5, 1.0, 1.5, 2.0))
)
ctrl <- makeTuneControlRandom(maxit = 5)
rdesc <- makeResampleDesc("CV", iters = 2L)

# socket mode ------------------------------------------------------------------

parallelStartSocket(2, level = "mlr.tuneParams")
#> Starting parallelization in mode=socket with cpus=2.
res <- tuneParams("classif.ksvm",
  task = iris.task, resampling = rdesc,
  par.set = discrete_ps, control = ctrl, show.info = TRUE
)
#> [Tune] Started tuning learner classif.ksvm for parameter set:
#>           Type len Def      Constr Req Tunable Trafo
#> C     discrete   -   - 0.5,1,1.5,2   -    TRUE     -
#> sigma discrete   -   - 0.5,1,1.5,2   -    TRUE     -
#> With control class: TuneControlRandom
#> Imputation value: 1
#> Exporting objects to slaves for mode socket: .mlr.slave.options
#> Mapping in parallel: mode = socket; level = mlr.tuneParams; cpus = 2; elements = 5.
#> [Tune] Result: C=2; sigma=0.5 : mmce.test.mean=0.0600000
parallelStop()
#> Stopped parallelization. All cleaned up.

# sequential -------------------------------------------------------------------

res <- tuneParams("classif.ksvm",
  task = iris.task, resampling = rdesc,
  par.set = discrete_ps, control = ctrl, show.info = TRUE
)
#> [Tune] Started tuning learner classif.ksvm for parameter set:
#>           Type len Def      Constr Req Tunable Trafo
#> C     discrete   -   - 0.5,1,1.5,2   -    TRUE     -
#> sigma discrete   -   - 0.5,1,1.5,2   -    TRUE     -
#> With control class: TuneControlRandom
#> Imputation value: 1
#> [Tune-x] 1: C=1.5; sigma=1.5
#> [Tune-y] 1: mmce.test.mean=0.0466667; time: 0.0 min
#> [Tune-x] 2: C=0.5; sigma=1.5
#> [Tune-y] 2: mmce.test.mean=0.0600000; time: 0.0 min
#> [Tune-x] 3: C=0.5; sigma=1.5
#> [Tune-y] 3: mmce.test.mean=0.0600000; time: 0.0 min
#> [Tune-x] 4: C=1; sigma=2
#> [Tune-y] 4: mmce.test.mean=0.0466667; time: 0.0 min
#> [Tune-x] 5: C=1; sigma=2
#> [Tune-y] 5: mmce.test.mean=0.0466667; time: 0.0 min
#> [Tune] Result: C=1; sigma=2 : mmce.test.mean=0.0466667

# multicore --------------------------------------------------------------------

parallelStartMulticore(2, level = "mlr.tuneParams")
#> Starting parallelization in mode=multicore with cpus=2.
res <- tuneParams("classif.ksvm",
  task = iris.task, resampling = rdesc,
  par.set = discrete_ps, control = ctrl, show.info = TRUE
)
#> [Tune] Started tuning learner classif.ksvm for parameter set:
#>           Type len Def      Constr Req Tunable Trafo
#> C     discrete   -   - 0.5,1,1.5,2   -    TRUE     -
#> sigma discrete   -   - 0.5,1,1.5,2   -    TRUE     -
#> With control class: TuneControlRandom
#> Imputation value: 1
#> Mapping in parallel: mode = multicore; level = mlr.tuneParams; cpus = 2; elements = 5.
#> [Tune] Result: C=2; sigma=1.5 : mmce.test.mean=0.0466667
parallelStop()
#> Stopped parallelization. All cleaned up.

^{Created on 2019-12-26 by the reprex package (v0.3.0)}

How to see intermediate results from tuning in mlr in parallel?

1 Answers1