Is nearest centroid classifier really inefficient?

Question

I am currently reading "Introduction to machine learning" by Ethem Alpaydin and I came across nearest centroid classifiers and tried to implement it. I guess I have correctly implemented the classifier but I am getting only 68% accuracy . So, is the nearest centroid classifier itself is inefficient or is there some error in my implementation (below) ?

The data set contains 1372 data points each having 4 features and there are 2 output classes My MATLAB implementation :

  DATA = load("-ascii", "data.txt");

#DATA is 1372x5 matrix with 762 data points of class 0 and 610 data points of class 1
#there are 4 features of each data point 
X = DATA(:,1:4); #matrix to store all features

X0 = DATA(1:762,1:4); #matrix to store the features of class 0
X1 = DATA(763:1372,1:4); #matrix to store the features of class 1
X0 = X0(1:610,:); #to make sure both datasets have same size for prior probability to be equal 
Y = DATA(:,5); # to store outputs

mean0 = sum(X0)/610; #mean of features of class 0 
mean1 = sum(X1)/610; #mean of featurs of class 1

count = 0;
for i = 1:1372 
   pre = 0;
  cost1 = X(i,:)*(mean0'); #calculates the dot product of dataset with mean of features of both classes
  cost2 = X(i,:)*(mean1');

  if (cost1<cost2)
    pre = 1;
  end
  if pre == Y(i)
    count = count+1; #counts the number of correctly predicted values
   end

end

disp("accuracy"); #calculates the accuracy 
disp((count/1372)*100);

score 0 · Accepted Answer · answered Apr 23 '17 at 12:57

There are at least a few things here:

You are using dot product to assign similarity in the input space, this is almost never valid. The only reason to use dot product would be the assumption that all your data points have the same norm, or that the norm does not matter (nearly never true). Try using Euclidean distance instead, as even though it is very naive - it should be significantly better
Is it an inefficient classifier? Depends on the definition of efficiency. It is an extremely simple and fast one, but in terms of predictive power it is extremely bad. In fact, it is worse than Naive Bayes, which is already considered "toy model".
There is something wrong with the code too
```
X0 = DATA(1:762,1:4); #matrix to store the features of class 0
X1 = DATA(763:1372,1:4); #matrix to store the features of class 1
X0 = X0(1:610,:); #to make sure both datasets have same size for prior probability to be equal 
```
Once you subsamples X0, you have 1220 training samples, yet later during "testing" you test on both training and "missing elements of X0", this does not really make sense from probabilistic perspective. First of all you should never test accuracy on the training set (as it overestimates true accuracy), second of all by subsampling your training data your are not equalizing priors. Not in the method like this one, you are simply degrading quality of your centroid estimate, nothing else. These kind of techniques (sub/over- sampling) equalize priors for models that do model priors. Your method does not (as it is basically generative model with the assumed prior of 1/2), so nothing good can happen.

Is nearest centroid classifier really inefficient?

1 Answers1