Formula for "Relative absolute error" and "Root relative squared error" used in machine learning (as computed by Weka)

Question

In open source data mining software Weka (written in Java), when I run some data mining algorithm like Linear regression Weka returns model and some model evaluating metrics for test data.

It looks like this:

Correlation coefficient                  0.2978
Mean absolute error                     15.5995
Root mean squared error                 29.9002
Relative absolute error                 47.7508 %
Root relative squared error             72.2651 %

What is the formula for "Relative absolute error" and "Root relative squared error"? I cannot figure that out. I would like to use this metrics to evaluate my own algorithms in Matlab.

Answer 1

From this presentation, in slide 22, and citing witten, here are the formulas:

Relative absolute error

Root relative squared error

with

• Actual target values: a1 a2 … an
• Predicted target values: p1 p2 … pn

Answer 2

The formula for Root Relative Squared Error is actually the formula for the Relative Squared Error. You need to take the square root of this formula to get what Weka outputs.

Answer 3

On page 177 of the Weka book Witten, Ian H., Eibe Frank, and Mark A. Hall. "Practical machine learning tools and techniques.“ Morgan Kaufmann (2005): 578, Relative squared error is defined as follows:

“The error is made relative to what it would have been if a simple predictor had been used. The simple predictor in question is just the average of the actual values from the training data. Thus relative squared error takes the total squared error and normalizes it by dividing by the total squared error of the default predictor.”

This is consistent with the Weka implementation. As a consequence, one needs the average of the targets on the train set to compute all the relative errors.