Sufficiently believable

Hey, here's something we haven't seen much of lately: a post about math. Yes, that is still my major and I haven't dropped out. That said, I'm pursuing a Philosopher's Doctorate, thus, rather than actually do any math, I'm just going to talk about it.

I was reviewing data reduction today and was struck by the tortured definition of a sufficient statistic: If T(X) is a sufficient statistic for θ, then any inference about theta should depend on the sample X only through the value T(X). That is, if x and y are two sample points such that T(x) = T(y), then the inference about θ should be the same whether X = x or X = y is observed.

Ok, the authors do give a formal definition that is more concise, but it still reflects this mentality that the world is this objective rules-based machine and we simply need to discover those rules. I don't know any theoretical physicists who believe that (and, yes, I do know several theoretical physicists; they aren't as rare as you might think). But the stats community, which serves primarily the empirical sciences (I'll be generous and include the social sciences in that group) where everything is debatable, still clings to this notion of absolute truth.

Of course, there are plenty of stats folks who don't. They're called Bayesians. Rather than get wrapped up around what θ is (or if it even exists), a Bayesian is only interested in what we believe about θ. A statistic is sufficient for θ if it fully informs our belief. That is, if our posterior belief about θ is the same whether we're given the statistic or the entire sample, then the statistic is sufficient.

The formal definitions drive this home.

Frequentist: A statistic T(X) is a sufficient statistic for θ if P(X|T(X)) does not depend on θ.

Baked into that is the notion of absolute truth. Otherwise, conditioning X on a function of itself makes no sense at all.

Bayesian: A statistic T(X) is a sufficient statistic for θ if P(θ|T(X)) = P(θ|X).

It's a fair debate, but it seems to me that if θ is the thing you care about but don't know, talking about sufficiency in terms of how much more you know after observing the statistic makes a lot more sense.

Just to be clear, Bayesians and frequentists don't actually disagree on what constitutes a sufficient statistic. The two definitions are equivalent. The disagreement is over what a sufficient (or any other) statistic actually tells you about the world.