7

When trying to fit Naive Bayes:

    training_data = sample; % 
    target_class = K8;
 # train model
 nb = NaiveBayes.fit(training_data, target_class);

 # prediction
 y = nb.predict(cluster3);

I get an error:

??? Error using ==> NaiveBayes.fit>gaussianFit at 535
The within-class variance in each feature of TRAINING
must be positive. The within-class variance in feature
2 5 6 in class normal. are not positive.

Error in ==> NaiveBayes.fit at 498
            obj = gaussianFit(obj, training, gindex);

Can anyone shed light on this and how to solve it? Note that I have read a similar post here but I am not sure what to do? It seems as if its trying to fit based on columns rather than rows, the class variance should be based on the probability of each row belonging to a specific class. If I delete those columns then it works but obviously this isnt what I want to do.

smci
  • 32,567
  • 20
  • 113
  • 146
G Gr
  • 6,030
  • 20
  • 91
  • 184
  • 1
    can you provide some information about your training_data and target_class type/structure/values? – Bee Nov 17 '12 at 05:23

1 Answers1

13

Assuming that there is no bug anywhere in your code (or NaiveBayes code from mathworks), and again assuming that your training_data is in the form of NxD where there are N observations and D features, then columns 2, 5, and 6 are completely zero for at least a single class. This can happen if you have relatively small training data and high number of classes, in which a single class may be represented by a few observations. Since NaiveBayes by default treats all features as part of a normal distribution, it cannot work with a column that has zero variance for all features related to a single class. In other words, there is no way for NaiveBayes to find the parameters of the probability distribution by fitting a normal distribution to the features of that specific class (note: the default for distribution is normal).

Take a look at the nature of your features. If they seem to not follow a normal distribution within each class, then normal is not the option you want to use. Maybe your data is closer to a multinomial model mn: 

nb = NaiveBayes.fit(training_data, target_class, 'Distribution', 'mn');
Bee
  • 2,472
  • 20
  • 26
  • Thanks, but how do you implement the mn model in matlab? For instance the documentation [here](http://www.mathworks.co.uk/help/stats/naivebayes.fit.html) doesnt give an example of its use? – G Gr Nov 17 '12 at 06:09
  • 1
    `nb = NaiveBayes.fit(training_data, target_class, 'Distribution', 'mn');` – Bee Nov 17 '12 at 17:00
  • You should include this in your answer! Thanks Bee. – G Gr Nov 17 '12 at 17:04
  • 1
    Accuracy is not the most "accurate" way of judging a model (pun not intended!) It can be heavily misleading in cases where there is small number of training sample, or a huge bias in one class vs another class. You have to remember that good models have to have balanced training information and lots of them. The bigger the training data, the closer your model is to the ideal model (which usually also means the less accuracy you are going to get because in real world things are not that black and white) – Bee Nov 17 '12 at 17:12
  • I'm still not quite sure what you are doing here but here is my two cents: It seems that you first apply clustering to your testing data and then pass the content of a cluster to your BN. It is possible to get an accuracy of 0% in this case if your features in the cluster are awfully close in the nD space and they all happen to fall into the incorrect class by your Bayesian Network. If you were dealing with more heterogeneous testing data, accuracy of 0% would be less likely. – Bee Nov 20 '12 at 20:41