Lehmann–Scheffé theorem

Suppose we don't just want a "really good" estimator of a parameter. Suppose we want the "best". There are several ways to quantify that. One of the most common is the Minimum Variance Unbiased Estimator (MVUE). That is, the estimate with a distribution that has a mean equal to the true value and a lower variance than the distribution of any other estimate with a mean equal to the true value.

The first thing we can observe from that characterization is that we are firmly back in frequentist land with this whole notion of the "true value" of the parameter. That said, the Bayesians have an equivalent structure, the Bayes Estimator, the which minimizes a loss function applied to the posterior distribution and, if that loss function happens to be the usual mean squared error, you wind up with pretty much the same thing. So, onward without philosophical debate.

How do we know when we have a MVUE? (Just to throw further ambiguity on that question, some authors call it a UMVUE, the leading "U" standing form "Uniformly".) Well, we need yet another definition.

A complete statistic, S, for parameter θ is one where, for every measurable function g such that E(g(S(X|θ))) = 0 for all θ, P(g(S(X|θ)) = 0) = 1 for all θ.

Wowsers. What's that all about? Well, let's step back from the trees and try to see the forest for a bit. Recall that a minimal sufficient statistic can be written as a function of any other sufficient statistic. What completeness is saying is that this statistic is so "full" of information about the parameter, that the only way to get rid of that information is to run it through a function that forces everything to zero. Why zero? Well, that helps with verification since a lot of times you wind up with intractable integrals in the expectation where all you can say is, "It comes out to something other than zero." All the rest of it is just mathematical formality to catch weird edge cases on sets of measure zero and the like.

Not that it matters much in practice, but for MVUE's, we can actually relax this to bounded complete statistics, where you only have to show the condition holds if g is bounded except on a set of measure zero.

You may have guessed that you have to work pretty hard to cook up a statistic that is minimally sufficient and not bounded complete. You'd be right. But, having defended against such nonsense, we can now move on to the result, the Lehmann-Scheffé Theorem:

If S is a bounded complete, sufficient statistic for θ and E(g(S(X)) = τ(θ), then g(S(X)) is a MVUE for τ(θ).

Don't get thrown off by the g and τ; they are just there to indicate that it's OK to run both the estimate and the parameter through further transformations. In particular, it's totally cool to come up with a statistic that meets all the criteria except for bias and then transform it to get rid of the bias.

Finally, the omission of the "minimal" condition on the statistic is not an oversight. Any statistic that is both complete and sufficient is also minimally sufficient.