Method of Moments and Satterthwaite's Estimator

This result should be near and dear to my heart since it's basically what saved my bacon on the CISS algorithm. Of course, whether that algorithm ever sees the light of publication is still an open question, but I digress.

The concept is very simple (which is why it's one of the oldest estimation techniques). You consider the moments of the random variable you're sampling, expressing them in terms of the parameter you care about. Then, you compute the actual moments of your sample distribution. Line them up and you have a system of simultaneous equations that you can use to solve for your parameters.

There are some downsides. The biggest is that such estimates may well be biased. For example, the method of moments applied to a Normal sample suggests the variance be estimated using the formula for population variance rather than sample variance. Not a huge deal if the sample size is reasonable, but a biased estimate nonetheless. Also, you may get a result that simply makes no sense.

An example of this (and how to deal with it) is Satterthwaite's Estimator. Rather than going deep into details that you can easily look up if you care, I'll focus on the flexibility of the method (and the corresponding responsibility on the practitioner).

Satterthwaite was trying to get the denominator of a t statistic. Basically, he wanted a linear combination of his random variables to be modeled as Chi-squared, but the degrees of freedom were unknown. Applying the method directly gives an estimator that works, but might go negative. Since there's no such thing as negative degrees of freedom, that didn't sit well.

Satterthwaite obviously wasn't the sort of guy who just shrugs it off and hopes it will work out. By working additional known constraints into the equations, he came up with another estimator that can't go negative. In fact, it's still pretty much the best one known and is still used today (that is, when it makes sense to solve for a tractable estimator and not just ram the data through a MCMC simulator).

The point is that, since most distributions have lots of moments and there are also other external constraints that can be considered, the Method of Moments is less of a method than a general framework for cooking up an estimator. There are lots of possible "right" answers. When you have very little information about the sampled distribution (as in CISS), it's a really powerful technique. But, it needs to be used carefully because the results might be just a bit nutty.