mlr: What is the best way to test for a FailureModel?

Question

The mlr function configureMlr() allows users to set the following parameter:

on.learner.error: What should happen if an error in an underlying learning algorithm is caught “warn”: A FailureModel will be created, which predicts only NAs and a warning will be generated.

What is the best way to check if a FailureModel has been returned? At the moment I am just checking the class of the model, and if it is not what it should be then I am assuming it is a FailureModel.

library(survival)
library(mlr)
library(mlrCPO)

data(veteran)
set.seed(24601)
task_id = "MAS"
mas.task <- makeSurvTask(id = task_id, data = veteran, target = c("time", "status"))
mas.task <- createDummyFeatures(mas.task)

preproc_pipeline <- cpoScale()  # Standardise the numerical data - center and scale
outer = makeResampleDesc("CV", iters=5, stratify=TRUE)  # Benchmarking

cox.lrn <- preproc_pipeline %>>% makeLearner(cl="surv.coxph", id = "coxph", predict.type="response")
learners = list( cox.lrn )  
bmr = benchmark(learners=learners, tasks=mas.task, resamplings=outer, measures=list(cindex), show.info = TRUE, models=TRUE)

model_id = 'coxph.scale'
mods = getBMRModels(bmr, learner.ids = c(model_id))
num_models = length(mods[[task_id]][[model_id]])

for (i in 1:num_models) {
  mod = getLearnerModel(mods[[task_id]][[model_id]][[i]], more.unwrap=TRUE)
  if (class(mod) == "coxph") {
    print(mod$coefficients)
  } else {
    print("Failure model")
  }
}

I tried the following,

  if (isFailureModel(mod)) {
    print("FailureModel")
  }

but got the error message:

Error in UseMethod("isFailureModel") : 
  no applicable method for 'isFailureModel' applied to an object of class "coxph"

I didn't get the notion that you connected your question to the code. The description of the arguments to `configureMlr` had three different possible bvalues that could be passed to `on.learner.error` and "warn" was not even the default. And you didn't attempt to set that parameter to something other than the default, and you didn't set up a situation that would have triggered an error. — IRTFM, Sep 11 '19 at 20:53

score 2 · Accepted Answer · answered Aug 28 '19 at 13:09

I don't think there is an easy solution to this (as least I am not aware of it).

Your approach does not seem to be far off from succeeding. However, as stated in ?mlr::isFailureModel(), it needs to be applied to a an object of class WrapperModel, not to an an object of a specific model class (e.g. coxph in your case).

mlr: What is the best way to test for a FailureModel?

1 Answers1