Plotting critical differences in R with imported data

Question

A critical difference (CD) plot for comparing classifiers over multiple data sets (Demšar2006) can be generated with the mlr package like this:

# THIS WORKS
library(mlr)
lrns = list(makeLearner("classif.knn"), makeLearner("classif.svm"))
tasks = list(iris.task, sonar.task)
rdesc = makeResampleDesc("CV", iters = 2L)
meas = list(acc)
bmr = benchmark(lrns, tasks, rdesc, measures = meas)
cd = generateCritDifferencesData(bmr)
plotCritDifferences(cd)

This requires the evaluation results to reside in a rather complex BenchmarkResult object, although the data is basically a matrix (where M[i, j] holds the score of classifier i for data set j). I have previously generated such data in a Python workflow and imported in R into a data.frame (as there seems to be no Python package for such plots).

How can I generate a CD plot from this data?

I thought about creating a BenchmarkResult from the data.frame, but didn't know where to start:

# THIS DOES NOT WORK
library(mlr)
# Here I would import results from my experiments instead of using random data
# e.g. scores for 5 classifiers and 30 data sets, each
results = data.frame(replicate(5, runif(30, 0, 1)))
# This is the functionality I'm looking for
bmr = benchmarkResultFromDataFrame(results)
cd = generateCritDifferencesData(bmr)
plotCritDifferences(cd)

Not sure what you're asking -- your code already generates a CD plot? The structure of the BenchmarkResult object is well-documented and you should be able to extract the information with only a bit of additional effort. — Lars Kotthoff, Dec 07 '16 at 18:33
I had to load package ‘PMCMR’ for the penultimate line to run. The plot looked fine to me then. — lawyeR, Dec 07 '16 at 19:43
@LarsKotthoff I added additional code to highlight my problem. Also, I'm looking into BenchmarkResult, but I'm still struggling to grasp R's object orientation, as I'm not very experienced with R. — rvf, Dec 09 '16 at 10:28
@lawyeR Thanks, the example code I provided runs for me as well. My problem is, that I can't plot data that I obtained from outside the mlr package. I added a second code block to clarify that. — rvf, Dec 09 '16 at 10:32
Have a look at the implementation of the `generateCritDifferencesData` function. It uses the raw data from the `BenchmarkResult` and should be relatively straightforward to adapt to other data. — Lars Kotthoff, Dec 09 '16 at 18:19
@LarsKotthoff Thank you, I could extract the necessary information from the code and was then able to create the plots. — rvf, Dec 12 '16 at 11:22

score 1 · Accepted Answer · answered Dec 12 '16 at 11:17

I finally managed to create the plot. It is necessary to set only a handful of the BenchmarkResult's attributes:

leaners with id and short.name for each classifier
measures
results with aggr for each dataset/classifier combination

The code may then look like this (smaller example of 5 datasets):

library(mlr)
# Here I would import results from my experiments instead of using random data
# e.g. scores for 5 classifiers and 30 data sets, each
results <- data.frame(replicate(5, runif(30, 0, 1)))
clf <- c('clf1', 'clf2', 'clf3', 'clf4', 'clf5')
clf.short.name <- c('c1', 'c2', 'c3', 'c4', 'c5')
dataset <- c('dataset1', 'dataset2', 'dataset3', 'dataset4', 'dataset5')
score <- list(acc)

# Setting up the learners: id, short.name
bmr <- list()
for (i in 1:5){
    bmr$learners[[clf[i]]]$id <- clf[i]
    bmr$learners[[clf[i]]]$short.name <- clf.short.name[i]
}

# Setting up the measures
bmr$measures <- list(acc)

# Setting up the results
for (i in 1:5){
  bmr$results$`dataset1`[[clf[i]]]$aggr <- list('acc.test.mean' = results[1, i])
}
for (i in 1:5){
  bmr$results$`dataset2`[[clf[i]]]$aggr <- list('acc.test.mean' = results[2, i])
}
for (i in 1:5){
  bmr$results$`dataset3`[[clf[i]]]$aggr <- list('acc.test.mean' = results[3, i])
}
for (i in 1:5){
  bmr$results$`dataset4`[[clf[i]]]$aggr <- list('acc.test.mean' = results[4, i])
}
for (i in 1:5){
  bmr$results$`dataset5`[[clf[i]]]$aggr <- list('acc.test.mean' = results[5, i])
}

# Set BenchmarkResult class
class(bmr) <- "BenchmarkResult"

# Statistics and plot
cd = generateCritDifferencesData(bmr)
plotCritDifferences(cd)

Anyone who could teach me better R to avoid these for loops and code duplication would still be very welcome!

Plotting critical differences in R with imported data

1 Answers1