Why linear transformation improves accuracy and efficiency of classification for high-dimensional data?

Question

Let X be an m×n (m: number of records, and n: number of attributes) dataset. When the number of attributes n is large and the dataset X is noisy, classification gets more complicated and the classification accuracy decreases. One way to over come this problem is to use linear transformation, i.e., perform classification on Y=XR, where R is an n×p matrix, and p<=n. I was wondering how linear transformation simplifies classification? and why classification accuracy increases if we do classification on the transformed data Y when X is noisy?

Answer 1

Not every sort of linear transformation would work, but some linear transformations are sometimes useful. Specifically, principal component analysis (PCA) and Factor Analysis are linear transformations often used for dimensionality reduction.

The basic idea is that most of the information is probably contained in some linear combination of the features of the dataset, and that by throwing the rest of them away, we are forcing ourselves to use simpler models / overfit less.

This isn't always so great. For example, even if one of the features is actually the thing we're trying to classify, it could still be discarded by PCA is it has low variability - thus losing important information.