How to plot per tree ROC curves from randomForest in R?

Question

I know that randomForest is supposed to be a black box, and that most people are interested in the ROC curve of the classifier as a whole, but I'm working on a problem in which I need to inspect individual trees of RF. I'm not very experienced with R so what's an easy way to plot ROC curves for the individual trees generated by RF?

Answer 1

I don't think you can generate a ROC curve from a single tree from a random forest generated by the randomForest package. You can access the output of each tree from a prediction, for example over the training set.

# caret for an example data set
library(caret)
library(randomForest)

data(GermanCredit)

# use only 50 rows for demonstration
nrows = 50

# extract the first 9 columns and 50 rows as training data (column 10 is "Class", the target)
x = GermanCredit[1:nrows, 1:9]
y = GermanCredit$Class[1:nrows]

# build the model
rf_model = randomForest(x = x, y = y, ntree = 11)

# Compute the prediction over the training data. Note predict.all = TRUE
rf_pred = predict(rf_model, newdata = x, predict.all = TRUE, type = "prob")

You can access the predictions of each tree with

 rf_pred$individual

However, the prediction of a single tree is only the most likely label. For a ROC curve you need class probabilities, so that changing the decision threshold changes the predicted class to vary true and false positive rates.

As far as I can tell, at least in package randomForest there is no way to make the leaves output probabilities instead of labels. If you inspect a tree with getTree(), you will see that the prediction is binary; use getTree(rf_model, k = 1, labelVar = TRUE) and you'll see the labels in plain text.

What you can do, though, is to retrieve individual predictions via predict.all = TRUE and then manually compute class labels on subsets of the whole forest. This you can then input into a function to compute ROC curves like those from the ROCR package.

Edit: Ok, from the link you provided in your comment I got the idea how a ROC curve can be obtained. First, we need to extract one particular tree and then input each data point into the tree, in order to count the occurances of the success class at each node as well as total data points in each node. The ratio gives the node probability for success class. Next, we do something similar, i.e. input each data point into the tree, but now record the probability. This way we can compare the class probs with the true label. Here is the code:

# libraries we need
library(randomForest)
library(ROCR)

# Set fixed seed for reproducibility
set.seed(54321)

# Define function to read out output node of a tree for a given data    point
travelTree = function(tree, data_row) {
    node = 1
    while (tree[node, "status"] != -1) {
        split_value = data_row[, tree[node, "split var"]]
        if (tree[node, "split point"] > split_value ) {
            node = tree[node, "right daughter"]
        } else {
            node = tree[node, "left daughter"]
        }
    }
    return(node)
}

# define number of data rows
nrows = 100
ntree = 11

# load example data
data(GermanCredit)

# Easier access of variables
x = GermanCredit[1:nrows, 1:9]
y = GermanCredit$Class[1:nrows]

# Build RF model
rf_model = randomForest(x = x, y = y, ntree = ntree, nodesize = 10)

# Extract single tree and add variables we need to compute class probs
single_tree = getTree(rf_model, k = 2, labelVar = TRUE)
single_tree$"split var" = as.character(single_tree$"split var")
single_tree$sum_good = 0
single_tree$sum = 0
single_tree$pred_prob = 0


for (zeile in 1:nrow(x)) {
    out_node = travelTree(single_tree, x[zeile, ])
    single_tree$sum_good[out_node] = single_tree$sum_good[out_node] + (y[zeile] == "Good")
    single_tree$sum[out_node] = single_tree$sum[out_node] + 1
}

# Compute class probabilities from count of "Good" data points in each node.
# Make sure we do not divide by zero
idcs = single_tree$sum != 0
single_tree$pred_prob[idcs] = single_tree$sum_good[idcs] /     single_tree$sum[idcs]

# Compute prediction by inserting again data set into tree, but read out
# previously computed probs

single_tree_pred = rep(0, nrow(x))

for (zeile in 1:nrow(x)) {
    out_node = travelTree(single_tree, x[zeile, ])
    single_tree_pred[zeile] = single_tree$pred_prob[out_node]
}

# Et voila: The ROC curve for single tree!
plot(performance(prediction(single_tree_pred, y), "tpr", "fpr"))