Determine importance of individual variables in WEKA

Question

I am trying to determine the importance of individual variables in the WEKA implementation of an LMT(Logistic Model Trees) DT (Decision Tree).

I would like to know the contribution that each individual variable has in a classification task, hence the need to determine the importance of each individual variable. This is to conduct a more in depth analysis of my results.

I have already looked into the "select attributes" tab, and corresponding algorithms (i.e. Principal Components, Info Gain, Ranker, etc.); however, these algorithms provide information regarding which combinatons or ranks of variables will contribute to the best (or most effective, or quickest, depending on your end goal, classifier).

However, I am not interested in ranking or selecting the most significant variables. I am interested in determining how much (in percentage form, for example) each variable contributed to the final classification score of my DT.

I have contemplated removing each variable one by one to determine how the score changes; but I am not sure that this can be manually done as the final score may depend on some underlying correlations, which is why I want to make this determination using all of the variables together (even if the contribution of one is zero).

So, the question is: Is there a way to measure the contribution of each INDIVIDUAL variable used in a classifier (even if that contribution is zero)?

Thanks in advance for any assistance.

Answer 1

Here is the answer from some inquiries and research. The fact is that the Information Gain is not a probability. The result of this attribute selector only provides information (an integer (0-1)) regarding the amount of information that a feature contributes to a "pure" classification.

For example, a feature with an InfoGain value of 1 means that all of the information available in that feature contributes to classification, though it does not mean that the use of that cue alone is able to conduct the entire classification.

As in many cases, classification is the result of correlations of different features (at least in my experience with Decision Trees). Therefore, an analysis of the path of each decision to arrive at classification is one way to detect the contribution of a cue through its correlations. The Information Gain algorithm assesses each feature or attribute as an individual entity, not taking into consideration other attributes that it combines with to make a decision on class membership.