OK, let's try this variance derivation again, this time remembering that probabilities need to sum to 1.
Recall that
μ = (nbk/2)∫θP(θ)dθ = E(X|block contains data)E(θ)
and that θ ~ Beta(a, b) where a is the number of blocks sampled that contained rows for the query and b is the number of blocks sampled that didn't.
Var(X|θ) = σ(θ)2 = E(|X - μ|2) = (1-θ)μ2 + θ[(nbk)2/3 - nbk μ + μ2] = (1-θ)p + θq
Var(X) = σ2 = ∫σ(θ)2P(θ)dθ
OK, we've been here before. Let's do the integrals right this time:
σ2 = ∫σ(θ)2P(θ) dθ = ∫[(1-θ)p + θq]P(θ) dθ
= ∫[(1-θ)p + θq]θa-1(1 - θ)b-1 dθ / B(a,b)
(forgot about that denominator last time - it's kinda important, especially when a and b start getting big.
= ∫(1-θ)bpθa-1 + θaq(1 - θ)b-1 dθ / B(a,b)
= [p∫θa-1(1-θ)b dθ + q∫θa(1 - θ)b-1 dθ] / B(a,b)
= [pΒ(a,b+1) + qB(a+1,b)] / B(a,b)
Here's the best part. Look what happens when you sub in B(a,b) = Γ(a)Γ(b) / Γ(a+b) and remember that Γ(z+1) / Γ(z) = z for all z ≥ 1:
= [pΓ(a)Γ(b+1) / Γ(a+b+1) + qΓ(a+1)Γ(b) / Γ(a+b+1)] / [Γ(a)Γ(b) / Γ(a+b)]
= (pb + qa) / (a + b)
The real world never comes out that cleanly. But, hey, you can check out the convergence graph yourself. Notice how the confidence bounds wander right up to the true value, but nearly always contain it. It really is that slick. There's much more testing to be done. My guess is that certain types of queries will obey these limits better than others. But, I don't think many will be too far off. The theory is sound and the implementation is correct. At any rate, it's coded and it's working on my test data. That's good enough for me to write this thing up.
Thursday, April 7, 2016
Wednesday, April 6, 2016
All done!!!
No April Fools this time; I found my mistake. I left the norming constant off the Beta distribution when I was integrating my variance over θ. That's the sort of mistake I often make. It actually simplifies things even more. I'll post the updated derivations tomorrow. Meanwhile, here it is, convergent with true confidence bounds:
A lot of writing still to do, but you could say I'm pretty happy about this.
Tuesday, April 5, 2016
Math as religion
Well, it's not like it's a new idea. While the ancient accounts are conflicting in the details, there is no question that the discovery of irrational numbers was a fairly big deal to the Pythagoreans of the fifth century BC and that it resulted in more than a few debates where dogma was valued over evidence.
As an ardent Christian, I'm not bashing religion (though I distinguish it from the higher calling of faith, which I would define as belief that transcends catechism). And, to be fair, Krushke's diatribe against frequentist hypothesis testing in my Bayesian stats book (which goes on for two full chapters) could be framed as a legitimate philosophical position rather than a statement of faith. But, that's not how he worded it.
I really have no problem with the fact that frequentists and Bayesians disagree on methods. Nor do I have a problem that the disagreement is not one of technique, but a fundamental understanding of what constitutes something as central as a random variable. What I don't particularly care for is that the arguments are always framed as "the other side is just nutty for not seeing how obvious this is." We're not talking about disenfranchised primary voters here. These are some of the greatest minds on the planet. The fact that there is disagreement should be cause for joy, not alarm. It means we've got a meaty topic on our hands. How is that not a good thing? Debate away, but let's show a little respect.
As an ardent Christian, I'm not bashing religion (though I distinguish it from the higher calling of faith, which I would define as belief that transcends catechism). And, to be fair, Krushke's diatribe against frequentist hypothesis testing in my Bayesian stats book (which goes on for two full chapters) could be framed as a legitimate philosophical position rather than a statement of faith. But, that's not how he worded it.
I really have no problem with the fact that frequentists and Bayesians disagree on methods. Nor do I have a problem that the disagreement is not one of technique, but a fundamental understanding of what constitutes something as central as a random variable. What I don't particularly care for is that the arguments are always framed as "the other side is just nutty for not seeing how obvious this is." We're not talking about disenfranchised primary voters here. These are some of the greatest minds on the planet. The fact that there is disagreement should be cause for joy, not alarm. It means we've got a meaty topic on our hands. How is that not a good thing? Debate away, but let's show a little respect.
Monday, April 4, 2016
Abstract
Here's a first draft of the abstract for this term's paper. Google's dictionary didn't recognize "heteroscedasticity" and wanted to change it to "heterosexuality". I don't need a spell checker making that mistake! I added it to my dictionary.
Financial Service Companies are now seeing data warehouse volumes running into the trillions of rows. Under such volumes, performance of queries using conventional “full search” technology (e.g., SQL, OLAP cubes) can degrade to where the user experience of immediate feedback is compromised. Simple random sampling methods do not yield reliable results due to the typically heavy-tailed distributions of financial data and heteroscedasticity. In this paper we present an algorithm that adjusts for both factors and returns both a point estimate and a confidence interval for the queried result. Empirical tests from a sample of projected cash flows are given to demonstrate the convergence and correctness of the results.
Financial Service Companies are now seeing data warehouse volumes running into the trillions of rows. Under such volumes, performance of queries using conventional “full search” technology (e.g., SQL, OLAP cubes) can degrade to where the user experience of immediate feedback is compromised. Simple random sampling methods do not yield reliable results due to the typically heavy-tailed distributions of financial data and heteroscedasticity. In this paper we present an algorithm that adjusts for both factors and returns both a point estimate and a confidence interval for the queried result. Empirical tests from a sample of projected cash flows are given to demonstrate the convergence and correctness of the results.
Sunday, April 3, 2016
Cart before the horse
Well, I've got a more or less working algorithm. That's a good thing. Now it's time to put the RE back in REsearch. I really need to do a more thorough search of the literature to make sure this thing is publication worthy. I've only turned up a few relevant papers so far, and nothing that would prevent CISS from getting published, but I certainly can't claim that I've looked very hard. So, lots of library work over the next couple weeks. Tedious, yes, but necessary.
One would normally do all this BEFORE developing the result.
One would normally do all this BEFORE developing the result.
Saturday, April 2, 2016
2012 Kettle Morraine 100
I've already posted the train wreck that was my first attempt at Kettle. This week's off-day throwback race report details round 2.
Run June 2-3, 2012
Oddly enough, I found it quite enjoyable. Not so much the part that involved staggering through the fields in the midday sun and then being iced down at each of the next three aid stations. But, what followed was without question my most remarkable recovery in any event, ever. The trip through the night was slow, but I could feel my strength coming back. By morning I was running again and even managed to salvage a top-10 finish. I've never pulled myself out of such a deep hole.
But, enough about last year. I decided to return to the Kettle Moraine 100 this year with the goal of actually getting it right. I've spent a lot of time reliving the nightmare of last year's return trip from the 50K turnaround and developed a strategy designed to avoid a repeat:
I still expect the night section, which is hillier and more technical to be significantly slower going than the day, but I'm hoping to be actually running it rather than walking like last year. In the process, I'd also like to move myself a few places up the finish order. A review of the start list casts some doubt on that secondary goal. Kettle is now part of the newly-formed Midwest Grand Slam and, the notable absence of course record holder Zach Gingerich notwithstanding, both the quality and, quantity of entrants are at record highs. Two hundred forty three folks sign up to tackle the 100 miles of glaciated trail with another eighty eight planning on stopping at 100K and seventeen teams in the relay.
Like last year, I pick up my number Friday afternoon and then go for a short jog on the Ice Age Trail to get a read on conditions. Simply put, they couldn't be better. The trail has just enough moisture to give a nice, springy landing without robbing any power and there isn't a trace of mud. I decide I'll run the first 100K in my road racing flats and then switch to my trail shoes for the rockier section at night. I then head an hour west to Madison to spend a very fine evening with my college cycling teammate Tom Rickner and his wife, Rebekah, who put me up for the night.
The first evidence of the larger field comes when I arrive at the start at 5:30AM to get one of the very last parking spots in the grass. I'm not sure where they are going to put everybody else, but I don't have time to worry about that. With all the extra commotion, the 30 minutes before the gun are barely enough to get my drop bags sent to the proper stations, pick up my timing chip, fill my bottles, and change into my racing flats. Seasoned racers will note the lack of a restroom trip in the preceding list. That would bother me in a shorter race, but I figure stopping at the first aid station in a 100 isn't a significant hit to the time.
Once underway, a lead group forms containing the usual nutballs who seem to have forgotten that the course distance has three digits. This is exacerbated by the fact that the conditions are very nearly perfect: 60F, low humidity, clear skies, excellent footing. Despite intentionally holding back, I still go through the first mile in 9 minutes, a minute faster than planned. I slow to what seems a truly absurd pace, and manage stay there. Steve Pollihan runs with me a bit, but it's clear he wants to go faster so I tell him to set his own pace.
I arrive at the Tamarack aid station at 5 miles in just under 50 minutes. I barely break stride as I've still got enough in my bottles to get me to Bluff at 7.5. Bluff is a longer stop as I make my requisite bathroom stop. It's 8 miles to the next manned station with just one water drop in between, so I top off my bottles and grab a little extra food.
This is the first section of true singletrack, the opening miles being on a grassy ski trail. I keep the "effort" (still feels ridiculously easy) constant which slips my pace a bit. Still, the increased technical demands of the trail have me more in my element and I pass a dozen or so runners. Fortunately, trail etiquette is observed and I'm never stuck behind anybody for more than a few seconds. Unfortunately, the stop at Bluff didn't get the job done and I need another restroom break by the time I arrive at Emma Carlin. That, and getting my extra water bottles out of my drop bag, makes it a seven-minute stop. I had resolved this year to not rush through aid stations (see odd-numbered strategy items above), but 15 minutes of down time in the first 16 miles is not what I had in mind, either.
It's nearly another 8 miles across the notorious fields and marsh to the next manned station at Route 67. While not nearly as hot as last year, I'm still glad I've got the extra bottles. The running is easy and I firm up the pace just a bit so I get out of the sun sooner. I leave my extra bottles at the Route 67 station and head back into the woods for the remaining distance to the turnaround at 50K. The lead relay team is tearing up the course; I meet their runner coming back while I still have nearly five miles to the turn. The first 100-mile solo runners start showing up while I still have over two miles to go. Not surprisingly, Tommy Doias is near the front. Before the start, he had indicated some concern about his fitness as he's coming back from injury, but he certainly appears to be running well now.
I hit 50K at 11:28 in 35th place; three minutes and 23 places behind last year. I hope the fine conditions have people taking this out too hard because, while the opening pace was very easy, I'm starting to feel the effort. Steve is at the aid station and asks how I'm doing. I reply that I'm a little concerned that I feel this tired given my pace. He echoes the sentiment.
The trail is quite busy on the return. Fortunately, I pass most of the field while still on the wide ski trail that leads in and out of the turnaround. By the time I hit the singletrack, the runners still heading out are moving pretty slow and happy to yield the trail. It's too early to be fighting for positions, but I do note that I am catching a few runners. At Route 67, I grab my extra bottles and head back out into the sun. It's nothing like last year, but it is getting warm. I'm considerably slower than the morning passage through this section. Still, it feels good to come striding into Emma Carlin with a bit of form as opposed to last year's drunken stagger. My legs are telling me I should forget about the big surge to sunset, but at least they are still willing to run.
I put my extra bottles back in my drop bag and also leave my shirt. It's soaked through with sweat which is causing some chafing and I'm sure I'll be back to the start before it gets cool. I decide I need to run easy for a few miles to recover from the fields. Leaving the station, I'm passed by a runner who I decide to name Jose Cuervo since he looks like he should be playing volleyball on a beach in Mexico rather than running through the kettles of Wisconsin (turns out his real name is Rolando Cruz). He's pretty much the opposite of me: young, broad shoulders, deep tan, and a full head of dark hair. His stride is smooth and powerful. I'm surprised anybody carrying that much muscle is running so well at nearly 50 miles. Less than a quarter mile into the trail, we pass a couple girls coming the other way. They wave happily to Jose; perhaps members of his support crew. As soon as they are out of sight, he decides running fast isn't such a happening thing. He smiles and offers a word of support as I pass, no doubt thankful that his little show of bravado was enhanced by my pasty-white, concave chest as a basis for comparison. I get a chuckle out of it myself and return the encouragement.
The rest of the way in is uneventful, through I'm never able to get back on pace. I'm a bit surprised that I don't see the first 100-mile solo heading out on the night section until I'm two miles from the start. Seems that the leaders have had to give up some pace as well. More surprising is the fact that I finish the 100K in 9th place (at 5:33PM). I certainly didn't pass 25 people coming in, so they must have been camping at aid stations or dropped out altogether. As I'm changing my shoes, Tommy comes over and informs me that I'm about to be 8th; he's not going out for the night section. He says his heel is OK, but he just doesn't have the miles in to run 100 today. Tommy was a big part of me finishing last year, pushing me out of aid stations and telling me I'd recover if I just kept moving. I think about returning the impetus, but decide to respect his decision. It's one thing to have difficulties, quite another to realize you aren't prepared.
The night section starts the same as the day, following the ski trails to Bluff. This is some of the easiest running on the course, but it's not feeling easy to me. In fact, I'm increasingly having difficulty running at all. I was half an hour slower coming back on the day section which is a pretty good indication the first 50K were too fast, regardless of how effortless they felt. Now it's looking like the wheels may be falling off entirely. It takes a full hour to get to the Tamarack station, 2 minutes per mile off the morning pace. Two runners pass me, and I usually take that seriously at this point in the race, but there's nothing I can do about it. As much as I want to put ground under me before sunset, it's obvious that I need to take a break and gather myself or risk having to walk the entire night. I walk for a mile eating and drinking along the way and then pick up a very easy jog into Bluff.
After Bluff, it's back onto the Ice Age Trail again, but this time the out and back is in the opposite direction, heading southwest to Rice Lake. I'm less than a mile into the trail when I hear footsteps coming up behind me. "Solo or Relay?" I ask, without looking back. "Fun Run!" comes a cheerful reply. Oh, no! Not that again!
The "fun run" (which, at 38 nighttime miles on technical trail, is itself a pretty stout ultra) is used to keep the trail warm since the 100 field gets very spread out. It's a good idea to have more people available to render assistance if needed, but I am really not in the mood to have dozens of fresh runners blowing by me over the next few miles. I ask my new companion why they didn't wait until 8PM to start and she says they had so many that wanted to run, they decided to start in waves to keep the trail from being clogged. That's something of a relief and it is nice to have some company. She runs with me for a couple miles and then moves ahead. Within minutes, another fun runner has latched on and when I offer to move over so he can pass, he says he'd rather just hang back for a bit as it seems I'm negotiating the trail pretty well.
And that is actually true. Sometimes the best way out of difficulties is simply to tend to something else. During the fifteen minutes of conversation, my form has returned. And, none too soon as we're into fairly technical terrain. My hope was to make it to the Route 12 aid station (77 miles) prior to sunset where I would get my headlamp out of my drop bag. I have my backup light in my pocket, but this trail has enough roots and rocks that the big light is definitely worth the extra weight. The last mile to the aid station is in fields, so we press the pace to get through the woods as the light dims. We get to the edge of the field just after sunset and give each other a quick high-5 for beating the dark. I tell him to go ahead as the surge has been felt and I'm going to need to run slow again for a bit. In the open field, there is plenty of twilight to get me into the station.
I take my time at the aid station getting a cup of chicken soup (which is about all my increasingly fragile digestive system can handle right now) while I hook up my light and battery. I'll need everything I've got left for what's to come. The four and a half miles from Route 12 to Rice Lake are the toughest of the course. There is some easy trail in the middle, but the rest is steep, uneven, and littered with rocks and roots. As expected, I have to walk most of the really steep stuff, both going up and down, but I'm pleased that I still have enough form to run the rest, albeit slowly. The round trip takes well over two hours, but it's done. I know I can run the rest and, while any hopes of a sub-20 are gone, this is shaping up to be a reasonable competitive result (it's impossible to tell who's in what event at night, but it seems like I'm still around 10th).
I make the stop as short as I can, opting not to revisit my drop bag. I seem to be teetering on the precipice of disaster and my body is interpreting even the briefest rest as a signal to start shutting down. The snap is completely gone from my legs, but I still have enough motor control to stay upright on the trail. My stomach is churning, but I'm able to get enough down and nothing is coming back up. My joints (particularly my hips, which are always a problem because of all the bike wrecks in my 20's) are stiff, but functional. In short, I feel like I've been running all day and as the clock rolls past midnight, that is pretty much the case.
I pass through Bluff for the fourth and final time and again find the ski trails to be tough going. I'm sure I'm moving faster than when I was on singletrack, but since they are much wider, it feels slower. I pass the Tamarack station getting nothing more than a cup of water. I'm pretty sure nothing else will stay down and there are only five miles to go. I shuffle along for another mile or so and then manage to find just a bit of form. It's not much, but with the finish so close, I decide to hang onto it and cover the rest of the distance without losing the pace.
I cross the line at 2:35 AM, for an official finish of 9th in 20:35:19. As is typically the case in ultras, the top 10 is jammed with folks in their 40's, so I don't get any age-group hardware. It's a little disappointing to lose a place in the last third of the race. That doesn't happen to me very often and it's never happened in a 100 before. I tend to view the late stages of an ultra as an annuity where I get to cash in on my patience earlier in the race. If I had been able to execute the last part of my plan and get back on it with 10 to go rather than 3, it might have worked out that way; 6th place was only 15 minutes ahead. It's still hard for me to believe that the opening pace was too firm, but I guess it was.
At any rate, that's a rather minor blemish on what was really a pretty solid run on a fantastic trail in great conditions. It's not odd at all that I found it quite enjoyable.
Run June 2-3, 2012
Oddly enough, I found it quite enjoyable. Not so much the part that involved staggering through the fields in the midday sun and then being iced down at each of the next three aid stations. But, what followed was without question my most remarkable recovery in any event, ever. The trip through the night was slow, but I could feel my strength coming back. By morning I was running again and even managed to salvage a top-10 finish. I've never pulled myself out of such a deep hole.
But, enough about last year. I decided to return to the Kettle Moraine 100 this year with the goal of actually getting it right. I've spent a lot of time reliving the nightmare of last year's return trip from the 50K turnaround and developed a strategy designed to avoid a repeat:
- Don't hang out, but make sure to get enough at each aid station.
- Absolutely no sustained efforts until past the fields on the return trip.
- Don't hang out, but make sure to get enough at each aid station.
- For that matter, no short efforts until past the fields, either.
- Don't hang out, but make sure to get enough at each aid station.
- Once past the fields, go ahead and get on it with the idea of getting as much technical trail done in the daylight as possible.
- Don't hang out, but make sure to get enough at each aid station.
- Once night falls, use the remaining technical stuff as recovery.
- Don't hang out, but make sure to get enough at each aid station.
- Get back on it for the last 10 miles (which are pretty easy trail).
I still expect the night section, which is hillier and more technical to be significantly slower going than the day, but I'm hoping to be actually running it rather than walking like last year. In the process, I'd also like to move myself a few places up the finish order. A review of the start list casts some doubt on that secondary goal. Kettle is now part of the newly-formed Midwest Grand Slam and, the notable absence of course record holder Zach Gingerich notwithstanding, both the quality and, quantity of entrants are at record highs. Two hundred forty three folks sign up to tackle the 100 miles of glaciated trail with another eighty eight planning on stopping at 100K and seventeen teams in the relay.
Like last year, I pick up my number Friday afternoon and then go for a short jog on the Ice Age Trail to get a read on conditions. Simply put, they couldn't be better. The trail has just enough moisture to give a nice, springy landing without robbing any power and there isn't a trace of mud. I decide I'll run the first 100K in my road racing flats and then switch to my trail shoes for the rockier section at night. I then head an hour west to Madison to spend a very fine evening with my college cycling teammate Tom Rickner and his wife, Rebekah, who put me up for the night.
The first evidence of the larger field comes when I arrive at the start at 5:30AM to get one of the very last parking spots in the grass. I'm not sure where they are going to put everybody else, but I don't have time to worry about that. With all the extra commotion, the 30 minutes before the gun are barely enough to get my drop bags sent to the proper stations, pick up my timing chip, fill my bottles, and change into my racing flats. Seasoned racers will note the lack of a restroom trip in the preceding list. That would bother me in a shorter race, but I figure stopping at the first aid station in a 100 isn't a significant hit to the time.
Once underway, a lead group forms containing the usual nutballs who seem to have forgotten that the course distance has three digits. This is exacerbated by the fact that the conditions are very nearly perfect: 60F, low humidity, clear skies, excellent footing. Despite intentionally holding back, I still go through the first mile in 9 minutes, a minute faster than planned. I slow to what seems a truly absurd pace, and manage stay there. Steve Pollihan runs with me a bit, but it's clear he wants to go faster so I tell him to set his own pace.
I arrive at the Tamarack aid station at 5 miles in just under 50 minutes. I barely break stride as I've still got enough in my bottles to get me to Bluff at 7.5. Bluff is a longer stop as I make my requisite bathroom stop. It's 8 miles to the next manned station with just one water drop in between, so I top off my bottles and grab a little extra food.
This is the first section of true singletrack, the opening miles being on a grassy ski trail. I keep the "effort" (still feels ridiculously easy) constant which slips my pace a bit. Still, the increased technical demands of the trail have me more in my element and I pass a dozen or so runners. Fortunately, trail etiquette is observed and I'm never stuck behind anybody for more than a few seconds. Unfortunately, the stop at Bluff didn't get the job done and I need another restroom break by the time I arrive at Emma Carlin. That, and getting my extra water bottles out of my drop bag, makes it a seven-minute stop. I had resolved this year to not rush through aid stations (see odd-numbered strategy items above), but 15 minutes of down time in the first 16 miles is not what I had in mind, either.
It's nearly another 8 miles across the notorious fields and marsh to the next manned station at Route 67. While not nearly as hot as last year, I'm still glad I've got the extra bottles. The running is easy and I firm up the pace just a bit so I get out of the sun sooner. I leave my extra bottles at the Route 67 station and head back into the woods for the remaining distance to the turnaround at 50K. The lead relay team is tearing up the course; I meet their runner coming back while I still have nearly five miles to the turn. The first 100-mile solo runners start showing up while I still have over two miles to go. Not surprisingly, Tommy Doias is near the front. Before the start, he had indicated some concern about his fitness as he's coming back from injury, but he certainly appears to be running well now.
I hit 50K at 11:28 in 35th place; three minutes and 23 places behind last year. I hope the fine conditions have people taking this out too hard because, while the opening pace was very easy, I'm starting to feel the effort. Steve is at the aid station and asks how I'm doing. I reply that I'm a little concerned that I feel this tired given my pace. He echoes the sentiment.
The trail is quite busy on the return. Fortunately, I pass most of the field while still on the wide ski trail that leads in and out of the turnaround. By the time I hit the singletrack, the runners still heading out are moving pretty slow and happy to yield the trail. It's too early to be fighting for positions, but I do note that I am catching a few runners. At Route 67, I grab my extra bottles and head back out into the sun. It's nothing like last year, but it is getting warm. I'm considerably slower than the morning passage through this section. Still, it feels good to come striding into Emma Carlin with a bit of form as opposed to last year's drunken stagger. My legs are telling me I should forget about the big surge to sunset, but at least they are still willing to run.
I put my extra bottles back in my drop bag and also leave my shirt. It's soaked through with sweat which is causing some chafing and I'm sure I'll be back to the start before it gets cool. I decide I need to run easy for a few miles to recover from the fields. Leaving the station, I'm passed by a runner who I decide to name Jose Cuervo since he looks like he should be playing volleyball on a beach in Mexico rather than running through the kettles of Wisconsin (turns out his real name is Rolando Cruz). He's pretty much the opposite of me: young, broad shoulders, deep tan, and a full head of dark hair. His stride is smooth and powerful. I'm surprised anybody carrying that much muscle is running so well at nearly 50 miles. Less than a quarter mile into the trail, we pass a couple girls coming the other way. They wave happily to Jose; perhaps members of his support crew. As soon as they are out of sight, he decides running fast isn't such a happening thing. He smiles and offers a word of support as I pass, no doubt thankful that his little show of bravado was enhanced by my pasty-white, concave chest as a basis for comparison. I get a chuckle out of it myself and return the encouragement.
The rest of the way in is uneventful, through I'm never able to get back on pace. I'm a bit surprised that I don't see the first 100-mile solo heading out on the night section until I'm two miles from the start. Seems that the leaders have had to give up some pace as well. More surprising is the fact that I finish the 100K in 9th place (at 5:33PM). I certainly didn't pass 25 people coming in, so they must have been camping at aid stations or dropped out altogether. As I'm changing my shoes, Tommy comes over and informs me that I'm about to be 8th; he's not going out for the night section. He says his heel is OK, but he just doesn't have the miles in to run 100 today. Tommy was a big part of me finishing last year, pushing me out of aid stations and telling me I'd recover if I just kept moving. I think about returning the impetus, but decide to respect his decision. It's one thing to have difficulties, quite another to realize you aren't prepared.
The night section starts the same as the day, following the ski trails to Bluff. This is some of the easiest running on the course, but it's not feeling easy to me. In fact, I'm increasingly having difficulty running at all. I was half an hour slower coming back on the day section which is a pretty good indication the first 50K were too fast, regardless of how effortless they felt. Now it's looking like the wheels may be falling off entirely. It takes a full hour to get to the Tamarack station, 2 minutes per mile off the morning pace. Two runners pass me, and I usually take that seriously at this point in the race, but there's nothing I can do about it. As much as I want to put ground under me before sunset, it's obvious that I need to take a break and gather myself or risk having to walk the entire night. I walk for a mile eating and drinking along the way and then pick up a very easy jog into Bluff.
After Bluff, it's back onto the Ice Age Trail again, but this time the out and back is in the opposite direction, heading southwest to Rice Lake. I'm less than a mile into the trail when I hear footsteps coming up behind me. "Solo or Relay?" I ask, without looking back. "Fun Run!" comes a cheerful reply. Oh, no! Not that again!
The "fun run" (which, at 38 nighttime miles on technical trail, is itself a pretty stout ultra) is used to keep the trail warm since the 100 field gets very spread out. It's a good idea to have more people available to render assistance if needed, but I am really not in the mood to have dozens of fresh runners blowing by me over the next few miles. I ask my new companion why they didn't wait until 8PM to start and she says they had so many that wanted to run, they decided to start in waves to keep the trail from being clogged. That's something of a relief and it is nice to have some company. She runs with me for a couple miles and then moves ahead. Within minutes, another fun runner has latched on and when I offer to move over so he can pass, he says he'd rather just hang back for a bit as it seems I'm negotiating the trail pretty well.
And that is actually true. Sometimes the best way out of difficulties is simply to tend to something else. During the fifteen minutes of conversation, my form has returned. And, none too soon as we're into fairly technical terrain. My hope was to make it to the Route 12 aid station (77 miles) prior to sunset where I would get my headlamp out of my drop bag. I have my backup light in my pocket, but this trail has enough roots and rocks that the big light is definitely worth the extra weight. The last mile to the aid station is in fields, so we press the pace to get through the woods as the light dims. We get to the edge of the field just after sunset and give each other a quick high-5 for beating the dark. I tell him to go ahead as the surge has been felt and I'm going to need to run slow again for a bit. In the open field, there is plenty of twilight to get me into the station.
I take my time at the aid station getting a cup of chicken soup (which is about all my increasingly fragile digestive system can handle right now) while I hook up my light and battery. I'll need everything I've got left for what's to come. The four and a half miles from Route 12 to Rice Lake are the toughest of the course. There is some easy trail in the middle, but the rest is steep, uneven, and littered with rocks and roots. As expected, I have to walk most of the really steep stuff, both going up and down, but I'm pleased that I still have enough form to run the rest, albeit slowly. The round trip takes well over two hours, but it's done. I know I can run the rest and, while any hopes of a sub-20 are gone, this is shaping up to be a reasonable competitive result (it's impossible to tell who's in what event at night, but it seems like I'm still around 10th).
I make the stop as short as I can, opting not to revisit my drop bag. I seem to be teetering on the precipice of disaster and my body is interpreting even the briefest rest as a signal to start shutting down. The snap is completely gone from my legs, but I still have enough motor control to stay upright on the trail. My stomach is churning, but I'm able to get enough down and nothing is coming back up. My joints (particularly my hips, which are always a problem because of all the bike wrecks in my 20's) are stiff, but functional. In short, I feel like I've been running all day and as the clock rolls past midnight, that is pretty much the case.
I pass through Bluff for the fourth and final time and again find the ski trails to be tough going. I'm sure I'm moving faster than when I was on singletrack, but since they are much wider, it feels slower. I pass the Tamarack station getting nothing more than a cup of water. I'm pretty sure nothing else will stay down and there are only five miles to go. I shuffle along for another mile or so and then manage to find just a bit of form. It's not much, but with the finish so close, I decide to hang onto it and cover the rest of the distance without losing the pace.
I cross the line at 2:35 AM, for an official finish of 9th in 20:35:19. As is typically the case in ultras, the top 10 is jammed with folks in their 40's, so I don't get any age-group hardware. It's a little disappointing to lose a place in the last third of the race. That doesn't happen to me very often and it's never happened in a 100 before. I tend to view the late stages of an ultra as an annuity where I get to cash in on my patience earlier in the race. If I had been able to execute the last part of my plan and get back on it with 10 to go rather than 3, it might have worked out that way; 6th place was only 15 minutes ahead. It's still hard for me to believe that the opening pace was too firm, but I guess it was.
At any rate, that's a rather minor blemish on what was really a pretty solid run on a fantastic trail in great conditions. It's not odd at all that I found it quite enjoyable.
Friday, April 1, 2016
All done!
April fools. Certainly very encouraging progress this week, but it's still not quite right. I ran it with several random seeds and this plot is pretty representative:
It seems to work great for driving out the big variability, but then it wanders around a bit. That doesn't bother me, but the fact that the confidence interval doesn't indicate as much bothers me a lot. In just about every trial, the confidence bounds exclude the real answer for an extended period between 300 and 400 blocks sampled. They generally do a pretty good job of everywhere else. I could just arbitrarily widen the confidence interval, but I'd like to know why I have to do that.
My first thought was that modeling the block sum given a hit as Uniform(0, max possible) was bogus. Turns out, it's not. I checked the data against that assumption and it came back looking pretty darn uniform.
My guess is that my test data set is simply too small. Since the total sum is just that, a sum, I'm counting on the central limit theorem to give me a normally distributed error. Between the fact that the sums in different strata have significantly different variances and the relatively small number of non-zero entries, things are still a bit more spread out. Also, we're sampling without replacement. Once more than half the data has been sampled, that's going to start cramping the whole "the block sum is an independent random variable" thing. This algorithm was conceived as one that would sample a sufficiently small percentage of the total to not worry about that.
Hooking this thing to the real data set will require a new data layer. I've architected it so that the data layer can be swapped out for something high-performance fairly easily, but there's still the non-trivial matter of writing a high-performance data layer and hooking it to a 4Tb data set that isn't really set up to be read this way.
In the mean time, I think I do have enough to start writing. The confidence intervals aren't way off. I think this could be presented as a first cut.
My first thought was that modeling the block sum given a hit as Uniform(0, max possible) was bogus. Turns out, it's not. I checked the data against that assumption and it came back looking pretty darn uniform.
My guess is that my test data set is simply too small. Since the total sum is just that, a sum, I'm counting on the central limit theorem to give me a normally distributed error. Between the fact that the sums in different strata have significantly different variances and the relatively small number of non-zero entries, things are still a bit more spread out. Also, we're sampling without replacement. Once more than half the data has been sampled, that's going to start cramping the whole "the block sum is an independent random variable" thing. This algorithm was conceived as one that would sample a sufficiently small percentage of the total to not worry about that.
Hooking this thing to the real data set will require a new data layer. I've architected it so that the data layer can be swapped out for something high-performance fairly easily, but there's still the non-trivial matter of writing a high-performance data layer and hooking it to a 4Tb data set that isn't really set up to be read this way.
In the mean time, I think I do have enough to start writing. The confidence intervals aren't way off. I think this could be presented as a first cut.
Subscribe to:
Posts (Atom)