mlr equivalent of carets model selectionFunction in R

Question

The caret library in R has a hyper-parameter 'selectionFunction' inside trainControl(). It's used to prevent over-fitting models using Breiman's one standard error rule, or tolerance, etc.

Does mlr have an equivalent? If so, which function is it within?

Answer 1

mlr only allows to choose optimal hyperparameters by optimizing certain measures/metrics.

However, essentially each "measure" in mlr is just a function that specifies how a certain performance is handled. You can try to write your own custom measure as outlined in this vignette.

Other than that, it might be worth opening this as a feature request in the new mlr3 framework, specifically in mlr3measures, since mlr itself is deprecated.

Answer 2

Posting an answer to my own question, I found this..

Estimate relative overfitting.

Source: R/relativeOverfitting.R

Estimates the relative overfitting of a model as the ratio of the difference in test and train performance to the difference of test performance in the no-information case and train performance. In the no-information case the features carry no information with respect to the prediction. This is simulated by permuting features and predictions.

estimateRelativeOverfitting(
  predish,
  measures,
  task,
  learner = NULL,
  pred.train = NULL,
  iter = 1
)

Arguments

• predish - (ResampleDesc ResamplePrediction Prediction) Resampling strategy or resampling prediction or test predictions.
• measures - (Measure list of Measure) Performance measure(s) to evaluate. Default is the default measure for the task, see here getDefaultMeasure.
• task - (Task) The task.
• learner - (Learner character(1)) The learner. If you pass a string the learner will be created via makeLearner.
• pred.train - (Prediction) Training predictions. Only needed if test predictions are passed.
• iter - (integer) Iteration number. Default 1, usually you don't need to specify this. Only needed if test predictions are passed.