Chicago Marathon

Sorry for the upfront spoiler, but I'm assuming that most people reading this already know that I had a good run at Chicago. The problem with good runs is that they make for rather dull race reports. So, since my first marathon was also Chicago 25 years ago, I thought I'd write a report on that train wreck with a few lessons learned applied to this year's effort. So, step into the way back machine, we're going to party like its 1993.

Even though my identity in my 20's was completely built around being a cyclist, I never took much joy in the usual cyclist practice of bashing runners. The truth is, I envied them. I had just turned 9 years old when Frank Shorter won the Olympic Marathon and I was instantly enamored with it. I tried so hard to be a runner. I was doing 60-mile weeks at age 12. But, genetics had other plans. By the time I was in High School, it was obvious that I simply was not put together like the really good runners. I could be good, but I'd never be really good and greatness was completely out of the question. I looked for sports more suited to my body type and found cycling. I didn't get great at cycling, but I did get really good. While it wasn't enough to live on, I had a license and a contract that said I was a professional athlete. I thought that was cool. Still, I wished I had been a runner. So, when I retired from cycling, I decided to run a marathon. 

I think I can be excused some hubris. Cycling is an endurance sport that uses predominantly legs, heart, and lungs. In the winter, I'd run to stay in shape and 10-15 miles runs were common. Anytime I'd enter a local race, I'd do pretty well. My 5K PR is a solid 16:20. I've heard about "hitting the wall", but I've also ridden dozens of cycling races that were five hours or longer and know how to manage effort. It seems like it won't take more than a few long runs to put a marathon effort together. My longest run prior to this is 15 miles. I extend that with a training runs of 18, 20, and 22 at around 8:30 per mile. I'm tired at the end, but not crushed. I decide that three and a half hours (8-minute miles) is a reasonable target. In retrospect, it probably was, but the devil is in the details.

In the intervening 25 years, I've observed a lot of details. I've also got pretty good at estimating my finish time. I have three workouts I use for that.

The first is the classic "Yasso 800s" named for Bart Yasso who originally published the relationship in Runner's World. It couldn't be simpler. Run 10x800m as fast as you can given that they all have to be roughly the same time (within a second or two). Your average time in minutes:seconds is the hours:minutes for your marathon. Like many others, I've found this predicts just a bit fast, so I always add five minutes. In August, I knock out ten at 2:54. That points to a 2:59 for this year's effort at Chicago.

The second is a baseline race; some race that I ran before another marathon where both the race and the marathon were good efforts. I then scale the time difference. I run the Big River 10 in September and finish third in 66:20. That's 20 seconds slower than three years ago when I won it outright and then ran a 2:58 at Milwaukee. Scaling that delta to a three hour effort again points to 2:59.

The final is "The Duck". It's a 3-mile tempo loop around Mallard Lake that I run pretty regularly so I have a long history of Duck times converted to performances in races. It takes a bit more discipline to get this one right because tempo is not "all out". I run this in late September and get a prediction of 3:01. Close enough; it appears I should be trying for a sub-3, but not much more than that.

These are all part of the larger 18-week training cycle leading up to the marathon which averages 90 miles a week, 35-40 of those going towards two or three quality workouts and the rest base (which, for me these days, is simply running to and from work).

My fiance, Kate, drives with me to Chicago. She's supportive, but makes no bones about the fact that she's really just happy to spend a night in a downtown Chicago hotel and go to a fancy restaurant. Check and check, though it taxes every bit of restraint in our waiter to not roll his eyes when we order a bottle of White Zinfandel. We walk back to the hotel in the darkness along the riverfront. There are moments so perfect you replay in your mind the rest of your life. This is one of them.

My wife, Kate, drives with me to Chicago. She's supportive, but makes no bones about the fact that she's really just happy to spend a night in a downtown Chicago hotel and go to a fancy restaurant. As I've found these things go better if we save the latter for after the race, I've booked two nights and made reservations Sunday at a place that doesn't have a White Zin on the wine list. We get an early dinner at Gino's East and call it a night.

I grew up just outside New York City and have spent almost all my adult life within a day trip of Chicago. It's interesting to observe how Chicago has both embraced and resented the "Second City" label it picked up when it passed Philadelphia in the 1890 census. The good folks of Cook County don't want to be New Yorkers, but they sure don't mind beating them. In 1993, Chicago is locked in a battle with the New York Road Runners Club to host the world's largest marathon and logistics are lagging the hype. This is before multiple waves, start corrals, or chip timing were a thing. You basically show up in the vicinity of the start line and wait for the gun to go off. As such, lining up next to the sign that has 3:30 on it means starting with a bunch of runners who have little hope of finishing under five hours. It takes eight minutes to get to the start line and another twelve to reach mile one. At 20 minutes per mile, this is going to take a really long time.

To get into the "A" corral at Chicago in 2018 as a guy means running a 2:50 (the standard is a bit more lax for women, presumably so the sub-elites have some chance of seeing each other). My 2:59:40 qualifier puts me in the B corral. It turns out that I could have petitioned to be moved up to the A corral at the number pickup the day before, but I'm not sure how anyone was supposed to know that. Anyway, with fewer than a thousand runners in the A corral, I'm still close enough to the front that I plan on gauging my effort by gun time rather than chip. Unfortunately, start procedures are still one of the few things Chicago doesn't do as well as Boston or New York. When the gun fires, we're held up by tape while the elites and then the A-corral head off. I guess this is to relieve congestion on the course, but it also means that there will be a significant difference between my gun and chip time. After about a minute they start releasing us, but the tape gets all wrapped around everybody. Nobody goes down but, in the confusion, I forget to check the time on the clock as I cross the line. I don't wear a watch, so I'll just have to assume that it was around a minute. There's not really much I can do with that information anyway. I get to the first mile at 8:05 and figure I'm probably just on the high end of the 6:50-7:00 I was targeting. Better slow than fast for mile one. I relax into the pace.

Miles two and three are closer to 10 minutes each. The road is jammed from curb to curb, but I've spent the last 15 years learning how to move up inside a pack of cyclists and find the same techniques work pretty well in running. Shortly after mile three, I find I have enough room to run my pace, but there's still the question of what pace that should be. A 3:30 isn't looking very likely, but I'd still like to be "comfortably" under 4 hours. I settle into a 7:00 pace and hold it for the next 10 miles, hitting the half in around 1:50. That's close enough to my original goal that I decide to get off the gas as my legs are starting to feel the effort. This doesn't alarm me in the least as big pushes early in the race are pretty common in cycling. Usually, it just takes a few minutes of soft pedaling to right matters. I'm about to find out that running doesn't work that way.

As I head towards the northern end of the course, two things concern me. One is that I keep forgetting the seconds from my previous mile split (was that 38 seconds from last mile or two miles ago?). I don't look at the minutes because I'm never off by that much. It seems like I'm holding things in the 6:45-6:50 range, but I worry I might be inadvertently crediting myself for an extra 10-15 seconds due to a missed split. Maybe there is some merit to wearing a watch in a marathon after all. The larger concern is that the pace feels just a bit firm. Not enough that I want to give up on it, but enough that it might get really ugly later in the race. At eight miles, the course turns south and suddenly everything feels easy. Apparently, we've been running into the wind.

I hit the half at 89:35 on the clock. ON THE CLOCK! I'm actually ahead of 3-hour gun pace! I look up the road and, sure enough, there's the 3-hour pace group for the A Corral. If I can stay near them the rest of the way, I'm golden.

Aside from spinning easy for a few miles, one thing I'd be doing to help bounce back from a hard start in a bike race would be grabbing something to eat. Unfortunately, I don't have anything on me and the aid stations only offer drinks. My easier pace brings up mile 15 in just under 2:10, but the legs aren't coming back. I'm going to have to dial it back even more.

I've tucked a couple gels in my waistband and pull one out and eat it. I've already thrown in two 6:40 miles just before the half. I had figured I could insert three such surges over the course of the race without destroying my legs for the final 10K. I put in one more to try to catch the back of the 3-hour group. I get to within 10 seconds at mile 15. I could try one more hard mile to close the gap, but I feel like I'm right on the edge and decide to go back to high 6:40's.

Things are not going well. My legs aren't coming back at all. In fact, they are completely going away. My stride is reduced to a shuffle. By 18, I'm barely holding 11 minutes/mile. At 20, I can see the finish line, but there's still a three mile out and back along the lake to go. The entirety of my clothing is a singlet, shorts, socks, and shoes. That was chilly, but fine at the 35-degree start but the temperature has dropped rather than warmed. It's now well below freezing and there's a howling wind off the lake. At least it's not sleeting. Oh, now it is.

The race organization describes the conditions for the 2018 race as "moderate". I guess I'd agree with that assessment. The temperature has been steady in the low 60's all run and there's been complete cloud cover. It was actually foggy prior to the start. That much humidity would normally be a problem, but we got a nice light rain for the second hour which mitigated fluid loss. I don't feel like I'm any lower than usual going into the final hour.

Just before 18, I make my biggest mistake of the race when I take an energy bar (I thought it was a gel) from an aid station. There's no getting solid food down at this point in the race so I should have just tossed it off. Instead, I try to jam it in my waistband. It turns out that getting things into the waistband pocket while running is a bit awkward. By the time I'm done, I notice that the people I was running with are about five seconds ahead of me. That doesn't sound like much, but it means that I'm now 15 seconds behind the three hour pace group. Even a fourth 6:40 surge won't catch them now. That means I'll not have the protection of a group when we turn back into the wind at mile 23.

I usually pace myself so that I expect to lose about 10 seconds per mile over the last 10K; I've found that results in a faster overall time than even splits. I hit 20 miles at 2:16:30 gun time. If I'm right about having a full minute in hand, that makes the sub-3 pretty much a done deal short of an injury or a complete meltdown (both are possible, of course). Sevens from here will make hitting the finish before the gun clock rolls over a squeaker. I decide every mile I can hold 6:50 is one more in the bank if I fade more than usual.

One of my favorite running authors, Alan Lawrence, writes that "There are few experiences worse than a marathon gone bad. The symptoms reported are those of terminal illness. Many real deaths are probably easier." As my pace continues to degrade, I find myself in complete agreement. I am determined not to walk. My stride is barely the length of my foot, but I manage a microscopic hop from one step to the next.

Coming back from the turnaround, Lake Shore Drive rises to become an elevated highway. It's not much of a climb, but it results in a pathetic 17-minute mile. It would be faster to walk, but I want to be able to say I ran the whole way, even if it is a rather generous definition of running. 

I don't hold on to 6:50 for long. I'm just under seven through 23, then give up a few more seconds as the course turns into the wind for the grind north to the finish. Most of the people around me have been dropped from the 3-hour group ahead and are struggling so the drafting opportunities are limited. The sense of urgency is palpable. Breaking three is pretty much the biggest prize out there for non-elite runners and we are right on the edge. In one of the things that makes running wonderfully unique among competitive endeavors, we spur each other on as best we can. The clock is the common enemy and we have no weapon to disrupt its advance, but we are united in our defiance.

By mile 24, The sleet has turned to snow and I can't even see the far end of Navy Pier. I tell myself I simply have to find a way to get the last 2.2 miles done in under half an hour. I can't take much more of this.

Mile 24 comes with 2:44:45 on the clock. I tell myself I simply have to find a way to get the last 2.2 miles done in under a quarter hour. A sub-3 on the gun clock is too good to pass up.

Since I'm on the elevated section of the highway, the finish is literally in sight, even though it's going to take a while to get there. It's enough to return a tiny amount of bounce in my step and I do finish out the race in about half an hour. I cross the line four hours and thirty four minutes after the gun. I didn't notice the seconds and I never bothered to look them up after results were posted.

I push for a few strides which results in a searing pain as one of my abdominals tears. I've never done that in a run before, but figure it can't be good. Surging to the line is not a happening thing today. I back off to 7:10 pace and the pain subsides. The one hill on the course comes right at 26 miles and that adds a few more seconds to my run in.

I hit the line with 3:00:35 on the clock, which I'm pretty sure gives me a sub-3, but I spend the next few minutes trying to get my phone out of my waistband so I can confirm that from the website. The site is obviously getting hammered right now and it takes a while to tell me that I've finished in 2:59:20, good for sixth place in my age group. It's my first (and likely to be only) top-10 in a World Major.

I shuffle through the finish area in a daze. Maybe someone put a medal around my neck; if they did, it's no longer in my possession. At the far end, I meet Kate who spent the morning perusing shops on Michigan Avenue until the weather got nasty. She says her feet are sore. I have no response that wouldn't result in cancelling the wedding, so I let it go.

The finish area is a happy place filled with runners that broke three, many for the first time. I collect my medal along with a rather generous (by big city marathon standards, anyway) amount of free food. I also get a text from Kate, who has been abandoned several miles away by an Uber driver who couldn't figure out how to get around the road closings. She says her feet are sore. This time, I have to laugh.

The variable in question

The blog is back for another school year. I did do a fair bit of research this summer, as well as a bunch of other interesting stuff at work (and took a nice vacation, too). However, I'm going to skip all that for the moment and get right to the question of the day because I'm discussing it with my adviser this evening.

We've been proceeding through all this analysis somewhat blindly assuming that it's a given that we should use the normed sample sum as our estimator. I give a brief (and, as it turns out, less than compelling) proof that this is the Uniform Minimum Variance Unbiased Estimator and then move on to the variance of the estimator, which is the real subject of the paper.

Not so fast on that first point. It's true that the normed sample average would be the UMVUE for the mean of an infinite population, but that doesn't translate as naturally as you might think to the population sum when things are finite. In fact, given what we know about the composition of the blocks, the estimator can actually be improved upon.

First, though, let's pretend we don't know anything about the composition of the blocks other than that they may contain partitions of correlated data. In that case, the sampled pairs of observations, that is, both the dependent and independent variables, are a sufficient statistic. We'll call this D and the realization of a sample will be the vector d=(x,y), the individual rows from all the sampled blocks. One important item to note here is that d is considered unordered. That is, we don't care which blocks got sampled first or where a row is placed within a block.

Now, we pick an arbitrary unbiased estimator. The first row in the sample times the population size will do, we call that S=Ny₁. We now use the Rao-Blackwellization process to make a better estimator by conditioning this on the observed sample, that is S' = E(S|D=d). I'll spare the algebra, because it's simple and I don't feel like typesetting it here, but it's pretty clear that, if ordering isn't important, the expected value of the first item in a finite sample, given the sample, is the sample mean. So, S' is the sample mean times the total number of rows in the population, which gets us right back to where we started. In fact, it's not hard particularly hard to prove that any unbiased estimator conditioned on the sample will give this result. So, our estimator is fine in this case.

HOWEVER...

That's not really our case. We have a lot more information available to us. When loading the data, we can pre-compute all sorts of things. We know the true sum for each partition (that is the sum when all rows from that partition meet the query criteria). We know how many rows are in each partition. We know which blocks those rows wound up in. We probably know even more than that if this isn't the first query we've ever received. We can track historical hit rates. Hit rates correlated to specific attributes in the query. Which partitions are particularly susceptible to variation based on certain query attributes. Heck, we might even know the actual answer; query processors frequently cache the last few queries they've returned because people have a tendency to hit "refresh" and there's no reason to go through the trouble of recomputing a result you just produced.

Why is this a problem? Well, it's not if you're happy with the naive estimator, but the point of this exercise is to return the tightest confidence intervals in the least time, so if we can produce a better estimator using it, we should.

Here's one quick example: Suppose we know that the data has only two partitions and one of those partitions is in a single block. We could randomly sample blocks, but we might wind up with a sample that excludes the single-block partition. That introduces extra variance to our estimator. On the other hand, we could always sample the single partition block, and then proceed with the rest of the sampling as in the uniformly-distributed case (which yields much tighter bounds). The resulting estimator will have less variance.

Yes, that's contrived, but the point is that the extra information can help. A lot. I'd like to leave that out of this paper because the point of this one was simply to demonstrate that we can create reliable confidence intervals for highly-correlated data. I think we've proven that both mathematically and empirically.

However, this plays really well into the research path I had set out for the rest of the dissertation. I now have a template for showing that we can not only assess the quality of our estimator, we can actually improve on it by leveraging the structure of the data. More importantly, we can change the structure of the data to make that lever even longer. As I argued when creating DBESt (which, by the way, is still a thing and I intend to use it to restructure the data), the optimal structure is an np-complete problem, so there's no point in trying to solve it. But, it can be approximated using either the Evolutionary Strategy of DBESt, or layering some sort of Dirichlet analysis on top of that to try to get rows from like distributions into the same blocks.

Bogus detection

This post is mainly for my adviser as i told him I'd have the implementation details section done tonight and may not get to prettying up this graph. Under the heading of "How can we tell if this is just plain wrong?" my thought was that a simple heuristic might be to not use a minimum sample size but rather a minimum non-zero sample size. That is, where we get in trouble is when we don't have enough non-zero blocks. So, I modified the simple block variance sampler to keep going until it had the specified number of non-zero blocks instead of at a fixed block count. The results are below:

As you can see, at really small sample sizes, this technique helps a lot (the horizontal scale is logarithmic, the points are 5, 10, 20, 50, 100, 250). That is, waiting for 5 non-zero blocks yields much better confidence intervals than simply stopping at 5 no matter what. However, by 10, it's that advantage has gone away. Somewhere between 5 and 10 non-zero blocks is enough that the method works.

So, this is a really simple heuristic: run a few queries using whichever method you want and plot the performance when the stopping rule is some number of total blocks versus the same number of non-zero blocks. When the two converge, that's your minimum non-zero count. Don't even test the width of the confidence interval if you don't have that many; it can't be trusted.

Something to show for it

Going dark for a week was intentional. It's the only way I could make any progress in light of current work demands. The good news is that I think I have made progress. Not on the Metropolis-Hastings stuff; I think that line of inquiry is dead for the moment. But, I found a better way to use the bootstrap that does almost as well (but not quite as well) as the full kernel.

That's a big win on several fronts in my eyes. First, it provides a nice segue from the bootstrap to the kernel. Second, the kernel, which is the most Mathy solution is still the winner. Finally, the full bootstrap, which is the more Data Sciency solution is the winner once you consider performance (it's about 50% faster than the full kernel). That sets up the next phase of research really well since D-BESt isn't very Mathy at all, but will be a great add to the bootstrap algorithm.

FWIW, here's the rejection graph (as always, 50 is the target).

One percent

Trying to finish this paper as we get into the busiest part of the year at work has pushed blogging a ways down the list of things I get done in a day. Most of what I've been doing on the paper has been the usual editing stuff that isn't that interesting to write about.

In less mundane news: I hit a few snags with the Metropolis-Hastings stuff and may just jettison that whole line of work if I can't get it fixed. It's really just a transitional step to the full kernel sampler, anyway. But, I haven't given up quite yet.

Looking ahead, I have formed a goal in my head as to what would constitute a useful result (from an applied standpoint, as opposed to just something that may or may not be mathematically interesting). I already know the D-Best is pretty good at cutting 90% of the data out of a query. I also know that CISS did a pretty good job of returning results on just 10% of the data read. So, now having the theoretical framework to justify those results, it seems reasonable to expect that we could produce good results reading just 1% of the data. Two orders of magnitude is no small thing in the real world. It would make our current $1.2-million cluster operate like one that costed $120 million. That's a number that shows up on the bottom line even for a Fortune-500 company like mine.

Granted, we already have other tuning methods that give us one order of magnitude, so it's really more like a $10 million difference. Still, I don't know any VP's that wouldn't take that if you offered it to them. (Though, my VP still grabs his chest and hyperventilates every time I suggest we actually write a production version of this stuff - I guess he's seen big ideas go down in flames before).

Actual contestants in the comparo

As I've noted before, BMH and BLB are train wrecks when the data is not independent. While I certainly want to cite those authors, I don't think it's fair to say that their algorithms are really what I'm competing against. I've modified them too heavily to work in the blocked sampling case.

So, who's actually in this contest? And by the way, what is the contest? The second question has an easy answer: find the best estimator of the block sum variance. The actual estimator of the sum is the same for all of these. But, we want to know how good it is. The "best" estimator isn't necessarily the one that comes the closest to the real value (though, that's a good start). The correlation with the sum matters, too. If the estimator and the block sum variance are positively correlated (they usually are), then you get too many rejections when both are low. So, it's quite possible to have an estimator that's quite accurate, but still produces bad confidence intervals too often (the sample block variance fails on this count).

1. Sample Block Variance: You want to know the variance of a population and you have a sample from that population? Well, just use the sample variance. It's easy to compute, unbiased, and has minimum variance. Too bad it sucks so badly in this case. The problem, as just mentioned is that it is very strongly correlated with the sample sum, especially at small sample sizes. Any stopping rule based on this estimator will stop too early too often.
2. Rho Variance: Rather than use the block sum sample variance, we derive the block sum variance as a function of the hit rate variance. We then use the sample variance of the hit rates and plug it into this function. Works great when the variance of the hit rate is really what's driving the block sum variance. Not so much otherwise.
3. Adjusted Rho Variance: Part of the problem with the above method is that there will be variations in the hit rate from one block to the next even if the actual parameter is constant. So, we compute the variance in the hit rate that we would expect to see if it really was constant and subtract that out (variances are nice that way, you can decompose them into pieces that can be added or subtracted as needed). This adjustment makes the method work well in both the above case and when things really distributed uniformly. Still doesn't help when the underlying distribution is changing, which is to be expected as the method doesn't even look at that.
4. Partition Bootstrap: To try to account for partitions that vary both in hit rate and distribution of the measures, we perform a bootstrap method where we resample the existing partitions to create blocks and then look at the variance of the empirical distribution of the blocks. I'm not entirely sure why this one doesn't work better other than the usual problem with correlation between the sample sum and variance estimate. At any rate, it's actually pretty bad across the board.
5. Metropolis-Hastings: Here, we try to construct the distribution of the entire population by estimating the parameters with an MCMC chain. We then sample these points and feed them into the exact formula for the block sum variance. I haven't finished validating my work on this one, but it appears to work fine. The only problem is that it's rather slow because you can't parallelize the MCMC chain (well, you can, but the results aren't as good). This algorithm borrows the idea from BMH that it's OK to run the chain on just a sample rather than the full data set. There's a variant that I'm also testing that the BMH authors suggested where you actually use a different subset of the data for each iteration of the chain.
6. Full Kernel estimator: Rather than generate an empirical distribution with an MCMC chain, we generate one using Kernel techniques. The resulting distribution is very similar to the MCMC distribution, as are the generally good results. It's still not as fast as the Adjusted Rho Variance, but it handles pretty much any case and the Kernel can be easily biased in the early going to avoid the problems of correlation with the sum estimator at low sample sizes. (I didn't find this to be necessary, but it's a nice feature to have).

So, 3 and 6 are the winners, though I'm pretty sure 4 would work fine if I put more effort into it. As 3 and 6 are the most original methods, I'm not really worried that 4 isn't doing great. If somebody else wants to publish an improvement, they are free to do so.

Lamest defense ever

I'm not saying the bible gives Peter a bad rap. He is, after all, called out as the first Pope which would presumably imply pretty high respect. But some of the anecdotes we read during Easter season are less than complimentary and (at least to me) needlessly so.

First we have Matthew quoting Jesus as calling Peter "Satan" when Peter suggests that maybe Christ dying isn't such a great plan. All four of the gospel writers note his betrayal during the trial. I've already written about John's little barb about him not being able to keep up running to the tomb. Today, we get as good an explanation as any as to why Paul became the primary defender of the gospel: Peter is just not a very good lawyer.

We tend to read the passage of the Pentecost through the lens of Renaissance paintings rather than constructing the scene in our heads directly from the text. The accusation makes a lot more sense if you take the story at face value. It starts by noting that the apostles, that is, twelve guys in their 20's, were all living in one house. Yeah, what could go wrong there? One morning they are all out front yelling out a bunch of stuff in a dozen different languages. The obvious conclusion is the one the bystanders came to: these dudes are drunk out of their minds.

Maybe Peter had never been to a frat party, but his defense is absurd: "We're not drunk, it's only 9AM." At what point after the discovery of alcohol did 9AM mean that young men are not drunk? Granted, he goes on to say some really important stuff after that and at least some folks were sold as the story ends with three thousand people getting baptized. But, seriously, that was your opening argument? That's the lamest defense ever.