I've been looking around, and can't seem to find an answer to this question:
If I train Naive-bayes to be a classifier on some data. Then I RE-USE this training data as the TEST DATA. Shouldn't I get 100% classification success? Thanks for reading!
Edit: It seems I've stimulated a discussion above my level of understanding. As such I don't feel it is up to me to take the role of 'accepting' an answer. However I am grateful for your input and will read all answer.
Actually, despite being accepted answer, @flyingmeatball's answer is (at least partially) wrong in this particular case. It describes related phenomenon, but clearly not the crucial one for the situation given.
What you described is a case, where you expect your model to have 100% training accuracy, and it does not. This has nothing to do with "data not expressing phenomenon enough" - this would be true for high generalization error, not training one.
Less than 100% training error means, that maybe data itself is too noisy to model (which flyingmeatball suggests), but actually, for training set this would be the case if and only if there are two exactly the same points with different labels. If this is not the case (and probably is not) the actual "problem" is that model that you selected has some internal bias. In simple terms - think about it as assumptions about the data, or even constraints, which your model will not change even if data clearly does not follow it. In particular, Naive Bayes have two such assumptions:
Features are independent, meaning that there is no correlation, no important link between label and more than a single feature at the time. If your features are wind and temperature, Naive Bayes will asume that it can make good decisions based on temperature itself, for example assuming that "good temperature is around 20 degress" and the same for wind, for example "at most 10km/h". It will fail to find relation based on both values, like "it is important for the temperature minus the wind to be at least 30", or something similar
It assumes particular distribution of values over each feature - usually this is MultiNomial distribution or Gaussian. These are nice families of distributions, but lots of features do not follow them. For example if your feature is "time at which people buy at my grocery store" (say you treat it as a continuos variable, measures exactly in some microseconds etc.), you will notice, that you have two "peak hours" one in the morning and one in the evening, thus Naive Bayes will do a horrible job fitting a single gaussian, which will have a peak at noon instead! Again, wrong assumption leads to wrong decisions.
Why do we do such assumptions then? Well, for many reasons, but one of them is because we care about generalization and not training, consequently it is a way of preventing our model from overfitting to heavily, at the cost of "underfitting" training set. This also helps dealing with noise, simplifies optimization and makes lots of other wonderful things :-)
Hope this helps.
No. Not all of the variability in the data may be explained in the features you have selected. Imagine you are classifying whether it is a good day to play tennis. Your features are temp, wind, precip. Those may be good descriptors, but you didn't train on whether there is a parade in town! On the parade days, the tennis court is blocked off, so even though your features should do a good job of explaining the known data, there are outliers that didn't fit the features.
Generally, there is randomness in data that you will not be able to capture 100%.
Updated per comments below
The question is whether training and testing on the same dataset will be 100% accurate, which I think we both agree will not work (they didn't ask what the assumptions of the NB were). Here is a sample dataset demonstrating the scenario above:
import pandas as pd
import numpy as np
from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
df = pd.DataFrame([[1,1,0],[1,0,0],[0,0,1],[1,0,1],[1,1,0]], columns = ['hot','windy','rainy'])
targets = [1,1,0,0,0]
preds = gnb.fit(df, targets).predict(df)
print preds
array([1, 1, 0, 0, 1])
Notice that the first case and the last case are identical, but the classifier missed the prediction for the last case. This is because the data at hand does not always perfectly describe accompanying classification. There are many other assumptions to NB,which could also describe cases where it fails, (which you excellently pointed out below) but my goal was just to come up with a quick demonstration that they would hopefully understand and would answer the question.
