How to tune the threshold for classification using Leave one out in MLR package?

Question

I am using mlr package in R to train my model. I wanted to train my model and do prediction. At the moment I use Recursive Partitioning for Classification and the sample is created using Leave one out method. I want to set the optimum threshold for the prediction.

I have tried using model, and set the threshold value to makeLearner, which did not work as expected.

I tried using MakeParam and tune predict.threshold in Learner, which also did not work properly.

library(mlr)

mlr_data <- as.data.frame(scale(dataset_correlation[,1:154]))
mlr_data$label <- factor(dataset_correlation$label)


task <- makeClassifTask(data = mlr_data,target = "label")
lrnr <- makeLearner("classif.randomForest",predict.type = "prob")
rdesc <- makeResampleDesc("LOO")

ps = makeParamSet(
  makeIntegerParam("predict.threshold", lower = 0.1, upper = 1.0)
)

ctrl = makeTuneControlRandom(tune.threshold = TRUE)
lrn2 = makeTuneWrapper(lrnr, par.set = ps, control = ctrl, resampling = rdesc)
r = resample(lrn2, task, rdesc, extract = getTuneResult)
print(r$extract)

Error:

"Error in checkTunerParset(learner, par.set, measures, control) :

Can only tune parameters for which learner parameters exist: predict.threshold"

I need the threshold to be tuned automatically and use that tune value for prediction.

Got the solution:

check this website https://www.analyticsvidhya.com/blog/2016/08/practicing-machine-learning-techniques-in-r-with-mlr-package/

Do you mean [classifier calibration](https://mlr.mlr-org.com/articles/tutorial/classifier_calibration.html)? — Lars Kotthoff, Sep 18 '19 at 16:40
No, I want the resampling to be done with a threshold tuned by TuneControlWrapper — Priyanka Mohan, Sep 19 '19 at 06:56
Still not sure if I understand. Something like the plots [here](https://mlr.mlr-org.com/articles/tutorial/performance.html#binary-classification)? — Lars Kotthoff, Sep 19 '19 at 14:50
Can't we tune the threshold like https://github.com/mlr-org/mlr/issues/856#issuecomment-214875911 ??? I want to tune it when I am doing prediction — Priyanka Mohan, Sep 19 '19 at 15:09
I tried using `tuneThreshold()` and tried adding `predict.threshold` in makeLearner to corresponding value. But then the misclassification rate was more. So can you suggest any other method? — Priyanka Mohan, Sep 19 '19 at 15:30
All of the error values you'll get are estimates and you won't get exactly the same on different splits of the data. That's by the very nature of machine learning. — Lars Kotthoff, Sep 19 '19 at 17:05
Great! Could you post this as an answer to your question please so that other people with similar problems can find it? — Lars Kotthoff, Sep 23 '19 at 14:57

How to tune the threshold for classification using Leave one out in MLR package?

0 Answers0