Epidemiology

So, what if you read yesterday's post and thought to yourself, "that's great, but I don't know about a field called Epidemiology"? Well, you could look it up on Wikipedia, of course, but, as long as you're going to trust some amateur hack to give you an encyclopedia entry, it might as well be me.

It started with shit. Seriously.

London in the 19th century was being overrun with the stuff. The flush toilet has been around in some form since ancient times. Sir Thomas Crapper contributed to its widespread development in the late 1800's, but did not actually invent the thing. What happened between ancient times and Crapper's fortune was the birth of Epidemiology.

In 1854, London was thrown into a panic as an epidemic of cholera hit the city. All sorts of things were blamed for it including the simple observation that the destitute were being hit the hardest, so it must just be that god doesn't like the lower class. Talk to Mr. Trump about Mexicans if you think we've progressed since then.

John Snow (no, not the dude from Game of Thrones) was a doctor in London who surmised that it might have something to do with the fact that people were literally drinking their own waste; the standard way to get rid of the stuff was to dump it out your window into the gutter where it could contaminate the water supply.

The powers that be were not impressed with his argument so he did what any self-respecting man of the Age of Reason would do: he gathered more data. He plotted where each victim lived on a map. He then looked for the nearest well. No luck; it appeared several wells must be contaminated and that would be quite a coincidence if the root cause was a breech allowing waste water into the fresh supply.

Then comes the brilliant part. Like most brilliancies, it's obvious in hindsight. People don't take helicopters to fetch water, they walk. So, he redrew his map, this time using minimum walking distance as his distance metric. He accounted for shortcuts like back doors and allyways. Blam. A single well was isolated. In the face of the newly presented evidence, officials issued an order to close the well in question. The epidemic ended as quickly as it had started and a new field of study was born. (And a new industry - London became the pioneering city for modern sanitation).

Like every field, Epidemiology has come a long way in the subsequent 150 years, but it's still basically about the same thing: extracting information embedded in disease and mortality data to suggest (not prove) a root cause. My master's work looked at methods to prioritize public health investigations based on the likelihood (or, unlikelihood) of the spatial distribution of disease cases. It was a minor result, for sure, but some very practical good came out of it. The New York State Department of Health used the results to streamline their investigation process and they did, in fact, turn up some nasty toxins.

Epidemiologists have been prying pearls of knowledge out of data for over a century, but the Data Science crowd seems to think they're inventing all this stuff. Read a typical Data Science paper and you'll see few, if any, references to journals like Biometrika (looks Russian, but it's actually an Oxford publication) or The American Journal of Epidemiology (the latter being where I was published). It's a gold mine. Just read all the significant results from the last 40 years in Epidemiology and you'll blow the doors off these newbies.

Please note that I'm not suggesting that the Data Science crowd is a bunch of idiots. They're just not using a lot of work that's already been done because it's been published in places they don't generally care about. In fairness, the vast majority of Epidemiology literature is very specific to particular applications so, unless you're in the trenches trying to sort out why diabetes rates are inverse correlated with income or some other such problem, you wouldn't bother reading it. However, if you want to get in on this Data Science thing (And, why not? It's a booming field), a review of the milestone articles is an easy way to get a leg up.