Gauss–Markov theorem

In statistics, the Gauss–Markov theorem (or simply Gauss theorem for some authors) states that the ordinary least squares (OLS) estimator has the lowest sampling variance within the class of linear unbiased estimators, if the errors in the linear regression model are uncorrelated, have equal variances and expectation value of zero. The errors do not need to be normal for the theorem to apply, nor do they need to be independent and identically distributed (only uncorrelated with mean zero and homoscedastic with finite variance).

The requirement for unbiasedness cannot be dropped, since biased estimators exist with lower variance and mean squared error. For example, the James–Stein estimator (which also drops linearity) and ridge regression typically outperform ordinary least squares. In fact, ordinary least squares is rarely even an admissible estimator, as Stein's phenomenon shows--when estimating more than two unknown variables, ordinary least squares will always perform worse (in mean squared error) than Stein's estimator.

Moreover, the Gauss-Markov theorem does not apply when considering more principled loss functions, such as the assigned likelihood or Kullback–Leibler divergence, except in the limited case of normally-distributed errors.

As a result of these discoveries, statisticians typically motivate ordinary least squares by the principle of maximum likelihood instead, or by considering it as a kind of approximate Bayesian inference.

The theorem is named after Carl Friedrich Gauss and Andrey Markov. Gauss provided the original proof, which was later substantially generalized by Markov.