1

I'm trying to resolve an error I'm facing with mlr::mergeBenchmarkResults, which is:

Error in mergeBenchmarkResults(bmrs = list(bmr, bmr_no_mos)): The following task - learner combination(s) occur either multiple times or are missing: 
* wnv_no_mos - rpart
* wnv_no_mos - rf
* wnv - rf_no_mos
* wnv - xgb_no_mos
* wnv - extraTrees_no_mos

Traceback:

1. mergeBenchmarkResults(bmrs = list(bmr, bmr_no_mos))
2. stopf("The following task - learner combination(s) occur either multiple times or are missing: \n* %s\n", 
 .     msg)

In a nutshell, my desire is...

I have two tasks, two learner sets, and two benchmarking objects. I wish to combine the two benchmarking objects using mlr::mergeBenchmarkResults.

tsk = makeClassifTask(data = wnv, target = "y", positive = "Infected")
lrns = list(makeLearner(id = "rpart", cl = "classif.rpart", predict.type = "prob"),
            makeLearner(id = "rf", cl = "classif.randomForest", predict.type = "prob"))
bmr = benchmark(learners = lrns, tasks = tsk, resamplings = rdesc, measures = meas, show.info = TRUE)

tsk_no_mos = makeClassifTask(data = wnv_no_mos, target = "y", positive = "Infected")
lrns_2 = list(makeLearner(id = "rf_no_mos", cl = "classif.randomForest", predict.type = "prob"),
            makeLearner(id = "xgb_no_mos", cl = "classif.xgboost", predict.type = "prob", nthread=25),
            makeLearner(id = "extraTrees_no_mos", cl = "classif.extraTrees", predict.type = "prob", numThreads = 25))
bmr_no_mos = benchmark(learners = lrns_2, tasks = tsk_no_mos, resamplings = rdesc, measures = meas, show.info = TRUE)

mergeBenchmarkResults(bmrs = list(bmr, bmr_no_mos))

What am I doing wrong?

Rahul
  • 2,579
  • 1
  • 13
  • 22
  • 1
    It looks like you have different tasks and learners in your individual benchmarks, so you can't really merge them -- a benchmark result contains results for all combinations, i.e. the learners from one benchmark result would also need to run on the task from the other benchmark result. This is what the error message is telling you. – Lars Kotthoff Feb 16 '19 at 21:47
  • Hmm. I guessed as much. I wish I could combine them though. There is a reason my tasks are different - they have different features and data prep methods. I wanted to compare across these tasks + learner combinations... – Rahul Feb 17 '19 at 14:35
  • You can still do so manually, it's just not quite as convenient as for benchmark results. – Lars Kotthoff Feb 17 '19 at 18:15
  • Gotcha. Thanks. – Rahul Feb 17 '19 at 23:30
  • I also faced the same problem several times already. Hope things will get easier in mlr. ;) – PhilippPro Feb 18 '19 at 09:38

0 Answers0