0

My understanding of classification accuracy always was "#correctly classified instances divided by #instances". Using Java-ML and applying LibSVM to a multi-label problem I get accuracies (and other measurements) for every CLASS. I can't figure out how they are related and what the overall accuracy is.

For example for my 3-class problem I get the following results:

Anger: Accuracy = 0.48148148148148145 | F = 0.35 | Precision = 0.310126582278481 | Error rate = 0.5185185185185185
Neutral: Accuracy = 0.9971509971509972 | F = 0.0 | Precision = NaN | Error rate = 0.002849002849002849
Surprise: Accuracy = 0.47863247863247865 | F = 0.5653206650831354 | Precision = 0.616580310880829 | Error rate = 0.5213675213675214

For which my code looks like this:

Map<Object, PerformanceMeasure> pm = cv.crossValidation(data, 5);
for (Object o : pm.keySet()) {
                System.out.println(o + ": Accuracy = " + pm.get(o).getAccuracy()
                        + " | F = " + pm.get(o).getFMeasure()
                        + " | Precision = " + pm.get(o).getPrecision()
                        + " | Error rate = " + pm.get(o).getErrorRate());
}
pscl
  • 3
  • 1

1 Answers1

0

An overall accuracy isn't that informative if there is not an even distribution of classes, which may be true in your case. You could still calculate one though if you wanted, see: http://spokenlanguageprocessing.blogspot.com/2011/12/evaluating-multi-class-classification.html

To answer your other question about how they are related, the results are being calculated on a class by class basis. So for neutral (sentiment?) you have great accuracy (over 99%) although you didn't predict a single neutral example (there may not be any in your dataset hence NaN). Your accuracy is being boosted because you are calling everything else "not neutral" and there are a lot of those cases. The other cases are easier to understand because they have positive and negative examples and more reasonable numbers. Hope this helps.

ozborn
  • 980
  • 5
  • 24