never2old4school: Variance 2.0

Tuesday, May 17, 2016

The derivation of the variance with the second parameter doesn't change too much, though the result is a bit messier. First we recall that we had carried a couple of constants p and q through the integration. Well, q is still a constant with respect to θ, but not ν. So, we re-write it as:

$q(\nu) =\nu^2/3 - \mu\nu + \mu^2$

This yeilds:

$\begin{align*} \sigma^2 &=\int \int \sigma(\theta,\nu)^2P(\theta)P(\nu)~d\theta d\nu \\ &=\int \frac{pb+q(\nu)a}{a+b}P(\nu)~d\nu \end{align*}$

This is going to get messy, so let's jettison the constant terms and focus on just the part dependent on ν:

$\begin{align*} \int q(\nu)P(\nu)~d\nu &=\frac{c+1-m}{(nb_k)^{c+1-m}- max\{x_i\}^{c+1-m}}\int (\nu^2/3 -\mu\nu + \mu^2)\nu^{c-m}~d\nu \\\\ &=\frac{(c+1-m)\left[\frac{\nu^{c+3-m}/3}{c+3-m} - \frac{\mu\nu^{c+2-m}} {c+2-m} + \frac{\mu^2\nu^{c+1-m}}{c+1-m} \right ]_{max\{x_i\}}^{nb_k}} {(nb_k)^{c+1-m}- max\{x_i\}^{c+1-m}} \\\\ &=\frac{\frac{1}{3}(c+1-m)}{c+3-m}\left( \frac{(nb_k)^{c+3-m}- max\{x_i\}^{c+3-m}}{(nb_k)^{c+1-m}- max\{x_i\}^{c+1-m}} \right ) \\\\ &\quad-\frac{\mu(c+1-m)}{c+2-m}\left( \frac{(nb_k)^{c+2-m}- max\{x_i\}^{c+2-m}}{(nb_k)^{c+1-m}- max\{x_i\}^{c+1-m}} \right ) \\\\ &\quad+\frac{\mu^2(c+1-m)}{c+1-m}\left( \frac{(nb_k)^{c+1-m}- max\{x_i\}^{c+1-m}}{(nb_k)^{c+1-m}- max\{x_i\}^{c+1-m}} \right ) \\\\ \end{align*}$

It's not quite that bad. The middle term is just μE(ν) and the bottom term is just μ². (Both those facts could be derived simply by looking closely at q(ν) rather than grinding out the integration.) The first term is the one of interest and it's the one that's going to drive down our variance estimate. But, it will do it in a controlled manner as the distribution pushes more mass towards the maximum observed block sum. Further, we can throttle it by tuning the value of c.

The rest of the variance is just symbol manipulation which I won't bother reproducing. Here's the final result (where the first term in the above result is renamed ν*):

$\sigma^2 =\frac{bp+a(\nu^*/3-\mu E(\nu)+\mu^2)}{a+b}$

Yes, I'm burying some computation in that formula, but not complexity. The terms all make intuitive sense; some of them just require a little number crunching. And, I do mean just a little. Twenty floating point operations seems like a lot until you compare it to reading several million bytes off a disk. Getting this variance right counts for a lot.

never2old4school

Tuesday, May 17, 2016

Variance 2.0

No comments:

Post a Comment