0

I'm new to data mining and Weka. I built a classifier with J48 in Weka using the GUI. When Weka is finished running it says:

Correctly Classified Instances 1035   -  68.543 %

Incorrectly Classified Instances 475  -  31.457 %

When running from my own code (C# using IKVM.NET), I re-evaluate my model with the same unlabeled data set, save the result of the predicted class into an ARFF file and count the results I get from 1500 records.

About 1300 records are correctly classified and 200 are not, which gives (I think) the precision measurement of 86%.

Is this correct? Why does the results differ then?

Felix Glas
  • 15,065
  • 7
  • 53
  • 82
Lorenzo
  • 673
  • 1
  • 11
  • 25

2 Answers2

5

I think your confusing "accuracy" and "precision", they are not the same thing.

  • Accuracy is the percentage of correctly classified instances for all instances
  • Precision is the percentage of correctly classified instances for those instances that have been classified as positive

In formulas:

  • Accuracy = (TP + TN) / (TP + TN + FP + FN) = #correct / #all_instances

  • Precision = TP / (TP + FP) = #correct_positive / #classified_as_positive

On the chance that you meant accuracy in both cases (accuracy of 1300/1500 is about 86%), there is no way to tell what is going on without seeing your code and the logs of the GUI, which would probably be too much for here.

Most likely explanation is that your code does somethings different than you do in the GUI. Maybe a different randomization, different split, different learning parameters, and so on.

Sentry
  • 4,102
  • 2
  • 30
  • 38
  • setting the classes in equal number (200 for class a, 200 for class b...) reducing the attribute number from 4000 to 700 i continue having a precision of 69% some says (i'm not talking about the stig) that deleting the attributes (words belonging to the dictionary in my case) that appear in too much istances of my data set will have increased my precision...so i calculate a measure for all the token in my dictionary like that: (number of comments in wich a token appear)/(total of comments) deleting the token with hightest measure should have increased precision instead i get 53% – Lorenzo Apr 15 '15 at 14:52
4

There are many statistical measures of performance

The result that Weka gives you are simply a measurement of the number of correctly and incorrectly classified records, i.e.

  • Correctly classified records = TP + TN
  • Incorrectly classified records = FP + FN

This is not the same as measuring precision. Precision is a measurement of how many of the correctly classified instances that are relevant, i.e.

  • Precision = TP / (TP + FP)

It is a highly specific metric that is best observed in relation to other metrics such as recall and accuracy. It may not be the best way to measure general performance of your model.

For more info about Precision and Recall see this: http://en.wikipedia.org/wiki/Precision_and_recall

Still, your numbers don't make much sense. However, not much more can be said without knowing more specifics.

greeness
  • 15,956
  • 5
  • 50
  • 80
Felix Glas
  • 15,065
  • 7
  • 53
  • 82