library(mlr3verse)
preformace_msr <- msr("classif.fbeta", beta = 1.5)
I am trying to use custom value of BETA in fbeta measure for classification model tuning.
But the above way of trying to give beta value throws an error in mlr3.
What is the right way of doing it in mlr3?
So the error was as follows:
library(mlr3verse)
preformace_msr <- msr("classif.fbeta", beta = 1.5)
#> Error: Cannot set argument 'beta' for 'MeasureBinaryimple' (not a constructor argument, not a parameter, not a field.
I think this has to do with the fact that only the F1-Measure (not F1.5-measure etc.) has been implemented in mlr3. See beta=1 in the source file.
In the mlr3 book you will find following under 6.3:
In this section we showcase how to implement a custom performance measure.
According to this section, you could implement an Fx-Measure with (I think my syntax is not the best/safest, but it should work):
library(mlr3verse)
library(mlr3measures)
library(R6)
# make custom measure:
MeasureCustomFbeta = R6::R6Class("classif.custom_fbeta",
inherit = mlr3::MeasureClassif,
public = list(
#declase field
beta=NULL, #delcare field
initialize = function(beta) {
self$beta <- beta
super$initialize(
# custom id for the measure
id = "custom_fbeta",
# required predict type of the learner
predict_type = "response",
# feasible range of values
range = c(0, Inf),
# minimize during tuning?
minimize = TRUE
)
}
),
private = list(
# customized scoring function operating on the prediction object
.score = function(prediction, ...) {
fbeta_cm = function(m, beta) {
pred_pos = sum(m[1L, ])
cond_pos = sum(m[, 1L])
if (m[1L, 1L] == 0L || pred_pos == 0L || cond_pos == 0L)
return(na_value)
P = m[1L, 1L] / pred_pos
R = m[1L, 1L] / cond_pos
((1 + beta^2) * P * R) / ((beta^2 * P) + R)
}
fbeta_cm(confusion_matrix(prediction$truth, prediction$response, prediction$positive)$matrix, self$beta)
})
)
# add it to the dictionary
mlr3::mlr_measures$add("classif.custom_fbeta", MeasureCustomFbeta)
Test:
# get data
data("Sonar", package = "mlbench")
# make task
task = TaskClassif$new(id = "Sonar", Sonar, target = "Class", positive = "R")
# make learner
learner = lrn("classif.rpart", predict_type = "response")
# predict
pred = learner$train(task)$predict(task)
pred$confusion
#> truth
#> response R M
#> R 87 16
#> M 10 95
# measure "classif.beta
measure_old <- msr("classif.fbeta")
pred$score(measure_old)
#> classif.fbeta
#> 0.87
# customized measure
measure_new <- msr("classif.custom_fbeta", beta=1.5)
pred$score(measure_new)
#> classif.custom_fbeta
#> 0.8801556
Measures did not have parameters until now. Will be solved in the next release via https://github.com/mlr-org/mlr3/pull/623.
