Delta method

Another biggie. I'm going to state it right up front without introduction because I, personally, am less interested in why it works than when it doesn't:

Let {Y_n} be a sequence of random variables such that sqrt(n)(Y_n - θ) converges in distribution to N(0, σ²). For a given function g and a specific value of θ, suppose g'(θ) exists and is not 0. Then: sqrt(n)|g(Y_n) - g(θ)| converges in distribution to N(0,σ²[g'(θ)]²).

Yes, that's a mouthful. All those caveats come from the fact that this result is derived using first-order Taylor series approximations. If any of them don't hold, the Taylor series doesn't work. It's true that these conditions do hold quite often and the Delta method is a very useful way to get a distribution on the transform of a sequence of random variables (typically, a transform of the mean). Still, let's take a closer look.

The first condition, that sqrt(n)(Y_n - θ) converges in distribution to N(0, σ²) is lifted straight from the Central Limit Theorem. You could derive similar results for any statistic that converged to some other distribution. The Delta method is targeted specifically at transforms of the mean. Fair enough.

The transform g has to be differentiable, at least at the point of interest. This point of interest, of course, is typically the sample mean. Also not a big deal.

The not zero condition, though, is a little troubling. Why would it matter that g'(θ) is zero? Working backwards from the result, it's obvious that the result is meaningless when g'(θ) is zero (the variance of the limiting distribution becomes zero). Still, what is it about that transform that causes such a problem?

This isn't some cooked-up theoretical case. Suppose the transform is something as simple as g(x) = x² and the point of interest is 0. Why shouldn't that work?

The problem is that first-order Taylor polynomials are kinda dumb. They only know a point and a slope. If the slope is zero, they degenerate to constants which doesn't make for a very interesting random variable.

So, here's the rub: what if g'(θ) is really close to, but not equal to zero? Technically, the Delta method works. In reality, not so much. Sure, it will converge if you give it a big enough sample size, but it will have to be a really, really, big sample. So, as with all of these things, applying the rule blindly is a quick route to some pretty bogus results. All these assumptions need to be checked.