MLR - Regression Benchmark Results - Visualisation

Question

What are the options for visualising the results of a benchmark experiment of regression learners? For instance, generateCalibrationData doesn't accept a benchmark result object derived from a set of regr. learners. I would like something similar to the calibration plots available for classification.

In response to @LarsKotthoff's comment, I (the OP) have edited my original post to provide greater detail as to what functionality I am seeking.

Edit:

I'm looking for actual vs predicted calibration type plots such as simple scatterplots or something like the plots that exist under Classifier Calibration. If I'm not mistaken, the following would make sense for regression problems (and seems to be what is done for Classifier Calibration):

• decide on a number of buckets to discretize the predictions on the x-axis, say 10 equal length bins (obviously you could continue with the breaks and groups interface to generateCalibrationData that currently exists)

• for each of those bins 10, calculate the mean "predicted" and plot (say via a dot) on the x-axis (possibly with some measure of variability) and join the dots across the 10 bins

• for each of those bins 10, calculate the mean "actual" and plot on the y-axis (possibly with some measure of variability) and join the dots

• provide some representation of volume in each bucket (as you've done for Classifier Calibration via "rag/rug" plots)

The basic premise behind my question is what kind of visualisation can be provided to help interpret an rsq, mae etc performance measure. There are many configurations of actual vs predicted that can lead to the same rsq, mae etc.

Once some plot exists, switching aggregation on/off would allow individual resampling results to be examined.

I would hope that the combination:

cal <- generateCalibrationData(bmr) 
plotCalibration(cal)

would be available for regression tasks, at present it doesn't seem to be (reproducible example below):

# Practice Data

library("mlr")
library(mlbench)

data(BostonHousing)
dim(BostonHousing)
head(BostonHousing)

# Define Nested Cross-Validation Strategy

cv.inner <- makeResampleDesc("CV", iters = 2)
cv.outer <- makeResampleDesc("CV", iters = 6)

# Define Performance Measures

perf.measures <- list(rsq, mae)

# Create Task

bh.task <- makeRegrTask(id = "bh",
                        data = BostonHousing, 
                        target = "medv")

# Create Tuned KSVM Learner

ksvm <- makeLearner("regr.ksvm")

ksvm.ps <- makeParamSet(makeDiscreteParam("C", values = 2^(-2:2)),
                        makeDiscreteParam("sigma", values = 2^(-2:2)))

ksvm.ctrl <- makeTuneControlGrid()

ksvm.lrn = makeTuneWrapper(ksvm, 
                           resampling = cv.inner,
                           measures = perf.measures,
                           par.set = ksvm.ps, 
                           control = ksvm.ctrl, 
                           show.info = FALSE)

# Create Tuned Random Forest Learner

rf <- makeLearner("regr.randomForest",
                  fix.factors.prediction = TRUE)    

rf.ps <- makeParamSet(makeDiscreteParam("mtry", values = c(2, 3, 5)))

rf.ctrl <- makeTuneControlGrid()

rf.lrn = makeTuneWrapper(rf, 
                         resampling = cv.inner,
                         measures = perf.measures,
                         par.set = rf.ps, 
                         control = rf.ctrl, 
                         show.info = FALSE)

# Run Cross-Validation Experiments   

bh.lrns = list(ksvm.lrn, rf.lrn)        

bh.bmr <- benchmark(learners = bh.lrns, 
                    tasks = bh.task, 
                    resampling = cv.outer, 
                    measures = perf.measures, 
                    show.info = FALSE)

# Calibration Charts

bh.cal <- generateCalibrationData(bh.bmr)
plotCalibration(bh.cal)

which yields:

> bh.cal <- generateCalibrationData(bh.bmr)

Error in checkPrediction(x, task.type = "classif", predict.type = "prob") : 

  Prediction must be one of 'classif', but is: 'regr'

> sessionInfo()

R version 3.2.3 (2015-12-10)

attached base packages:

[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:

[1] mlbench_2.1-1     ROCR_1.0-7        gplots_3.0.1      mlr_2.9           
[5] stringi_1.1.1     ParamHelpers_1.10 ggplot2_2.1.0     BBmisc_1.10      

loaded via a namespace (and not attached):

 [1] digest_0.6.9        htmltools_0.3.5     R6_2.2.0            splines_3.2.3          
 [5] scales_0.4.0        assertthat_0.1      grid_3.2.3          stringr_1.0.0          
 [9] bitops_1.0-6        checkmate_1.8.2     gdata_2.17.0        survival_2.38-3        
[13] munsell_0.4.3       tibble_1.2          randomForest_4.6-12 httpuv_1.3.3           
[17] parallelMap_1.3     mime_0.5            DBI_0.5-1           labeling_0.3           
[21] chron_2.3-47        shiny_1.0.0         KernSmooth_2.23-15  plyr_1.8.4             
[25] data.table_1.9.6    magrittr_1.5        reshape2_1.4.1      kernlab_0.9-25         
[29] ggvis_0.4.3         caTools_1.17.1      gtable_0.2.0        colorspace_1.2-6       
[33] tools_3.2.3         parallel_3.2.3      dplyr_0.5.0         xtable_1.8-2           
[37] gtools_3.5.0        backports_1.0.4     Rcpp_0.12.4