Centered

Well, my intent was to write up the eigenvalue stuff for Data Mining today but the production support work yesterday messed up my schedule a bit. I ended up taking today pretty much off rather than stress myself out this early in the semester. It's now 9:30PM and I should get to sleep so I'll just drop a little pearl that anybody who's taken a stats course has probably heard, but the application of it doesn't become clear until you start working with large data sets.

Data can always be centered. That center may be very artificial, but you can always transform it so that it does have a center. In the vast majority of cases, the sample mean is used. Of course, one could just as easily use the median or the mode. Why the mean? It comes down to how we measure variability.

The variability of data is usually measured by using a sum of squares. There are good reasons to do this when you get to higher dimensions. If you're projecting a set of points in, say, 3-space, onto a 2-dimensional line, the projection lies along the line and the error is a perpendicular vector. As even the scarecrow can tell you at the end of The Wizard of Oz, perpendicular projects mean you square the sides to get the magnitudes to add up.

So, since the point of a center is to minimize the variability between the data and the center, you solve for the minimum sum of squares and that turns out to be the sample mean. If, instead, you just used linear distance rather than squared distance and minimized that, you'd get the median. And, if all you cared about was whether things were equal or not (that is, your distance from the center is zero if you are at the center and 1 if your not) then your minimum variability is around the mode.

It's hard to see why this matters in a stats class. Everybody knows we're going to use the mean. But, when analyzing big data sets, your distance measure may be all sorts of weird things. Certainly, the 0/1 measure that yields the mode is a very common thing to care about and finding the most frequent value (i.e., the mode) is a very common database operation. Medians also get used a lot; knowing the point that cuts your data in half is useful in many situations. It's helpful to keep in mind that the answers aren't random - they result from the conscious choice of how to measure things.

The other side of Big Data

This will sort of be my off-day post because it's sort of an off day. I worked late last night and some more today monitoring some jobs that run long.

I'm currently the subject in a trial that, presumably, has some Big Data implications (though I have no idea what those are). For obvious reasons, the research department at work is interested in fitness trackers. My understanding is that the correlation between fitness and life expectancy is very weak (the quality of life may be better, but actual longevity is what a Life Insurer cares about). The hope is that some better correlating variables can be found with the use of the latest generation of fitness trackers.

So, lot's of folks at work are wearing fitness trackers for the next month and a half. I got a Basis Peak. On the surface, it's an exceedingly cool little device. The screen is like that of a smartwatch (actually, since it pairs to your phone, I guess it qualifies as a smartwatch). The user interface is reasonably intuitive and the touchscreen is a nice mix of responsive without being twitchy.

Underneath the skin, I'm less convinced. On my run this morning, it had my heart rate varying all over the place. In reality, it was quite steady (as one would expect on an easy run with a group). The distance measured was also a bit short. It did seem to do a pretty good job of counting my strides. It collected lots of other data, too, but I have no idea how to interpret that.

Truth is, I don't really care. I know what running easy is supposed to feel like. I know what tempo pace feels like. I know what marathon pace feels like. I really don't need a device of any kind to run a desired pace. A few years back, I ran the St. Louis Track Club pace series. It's 16 "races" where your finish time is the absolute difference between your projected and actual time. In other words, if you say the 5K will take you 19:30 and you run it in 19:45, your "finish" time is 15 seconds. Obviously, you're not allowed to wear a watch during the run. My total for the series (best 3 out of 4 in a month count, so 12 total) was under two minutes. And if you think that's good, yes, it was (third overall if I recall correctly), but the series winner was half that. Good runners have pacing dialed in pretty tight.

Anyway, some of the variance I was seeing could well be user error. I just started using this thing yesterday and may well not have the strap tight enough or something else set up wrong. Regardless, it's information I just don't need and I'm not really a gadget geek so, cool as it is, I won't be running out and buying one when this study is over.

Now, what I would like to know is what they are doing with the data? Of course, it's anonymous (and, I wouldn't care if it wasn't - there's nothing about my fitness that I need to keep secret). What I mean is how is this going to help them make more money on life insurance My guess is that, since people are so willing to post this stuff on publicly available sites, they want to make special rate offers to folks they can identify as very good risks. It's illegal to ask for this sort of information, but if somebody wants to volunteer it, using it is fair game. Google is doing the same thing getting into car insurance (hint: if you get pissed at the driver who cuts you off, don't post that on Facebook; your insurance company WILL find out about it sooner or later and classify you as having potential for road rage).

It's far enough from my thesis topic that my interest is purely curiosity, but it's still a question I will bother to ask.

More graphing in R

Apparently, our graphs are supposed to look nice. I suppose that does matter sometimes. I had to monitor a process all evening, so I spent it messing around with R's many and varied graphing packages trying to make my homework assignment pretty. Here's an example using just the base package.

#probability and cummulative functions
p = function(x) 6 * x * (1 - x)
cdf = function(x, width) cumsum(p(x)*(width/length(x)))

#take the midpoint of 100 even-spaced intervals
x = ((0:99) + 0.5)/100

#plot the pdf
plot.new()
plot(x, p(x), type="l", col="blue", ylab = " ")

#plot the cdf with a reference line to show it goes to 1
lines(x, cdf(x, 1), type = "l", col="red")
lines(c(0,1), c(1,1), lty=2)

#pretty it up a bit
title("p(x) = 6x(1-x)")
text(0,1.3, "p(x)", pos=4, col="blue")
text(0,1.1, "cdf(x)", pos=4, col="red")

Oooh and aaaah (though I have no idea why blue and red saved to very dark blue and maroon - they were quite bright when displayed in RStudio).

Small steps revisited

As I mentioned as part of The Ultrarunner's Guide to Long IT Projects, small steps is the way to get things done when you're in it for the long haul (actually, it works pretty well all the time). It seems my advisor agrees. We met today on what I would do to make the Bayesian class a true graduate course and decided we'd look for a piece of my thesis topic that could be tackled as a little chunk.

The most promising approach seems to be to try to solve a much more specialized case of the problem. Specifically, I'm going to look at how sampling affects some individual queries that that we run at work. We'll tune them for both performance and accuracy and see if we can't beat the regular cube. Most importantly, we'll look at whether our confidence intervals (or, more correctly, our posterior distribution on the result) matches empirical results. Even in this reduced form, this is not an easy task, but at least it's well defined.

Another avenue that occurred to me while we were tossing out ideas was the thought that rather than waiting for the posterior distribution to sufficiently converge, we could put a cost function on the error and then use hitting the expected value of that cost as our stopping rule (that is, keep sampling until the expected cost drops below a pre-set threshold.)

At any rate, I'm pretty excited to actually have some direction. Up until now, it's been lots of ideas but no clear path on how to proceed. I'm willing to put the ideas down for a bit in exchange for making some real progress.

Graphing in R

OK, I'll give it that much: graphing in R is trivial. I took an example diagram from the class text (cumulative proportion of coin tosses coming up heads) and tried to replicate it. Even this code is pretty verbose; you could jam all the graph commands into a single call. But, I don't like going for super-concise code as it's too hard for the next person to figure out what you did.

CummFlips = function(n, p)
  cumsum(rbinom(n, 1, p)) / (1:n)

plot.new()
plot.window(c(1,500), c(0,1), log="x")
plot.xy(xy.coords(CummFlips(500, .5)), "o")
axis(1)
axis(2)
title("Running Proportion of Heads",NULL,
  "Flip Number","Proportion")

From what I've sorted through online, the R crowd isn't nearly as fanatical about sticking to the functional roots in the language as are the adherents of it's progenitors. I think a lot of R practitioners would have put the binomial vector in a variable rather than using nested functions. As a C# programmer, I certainly have no problem with using variables, but it doesn't seem necessary and if you can't even get past an exercise this simple without trashing the language design philosophy, you might want to move on to something else.

The graph (at least the left hand portion) obviously changes each time you run this. Here's an example output:

Missing observations

Data cleansing was the subject of today's lecture for Data Mining. We spent most of the time looking at what to do in the case of missing observations. Granting that filling in a few values here and there is probably simplifying things without biasing results, I'd say the strategies offered were pretty bogus. I've held for some time that Data Science is just Statistics without the rigor and this bag of unsupportable tricks didn't do much to dissuade me.

I'm pretty sure that part of the problem is simply that the techniques that have some merit are beyond the level of this course (it's a mixed grad/undergrad course so, while we PhD types will be expected to produce some meaty theoretical work, the lectures are tuned to the undergrads). That said, there's at least one strategy for missing observations that I think a lot of CS folks are too quick to dismiss: go get the data.

Yes, it's a pain. I spent hundreds of hours pouring through motor vehicle records and mortgage papers preparing the data set I used for my masters work. Not because I needed a clean data set for the thesis, but because the work was actually being used by the New York State Department of Health to direct cancer investigations. Sure it, would have been easy enough to just fill in some data, but there are times when getting the answer right really matters.

You could certainly argue that many of today's data sets are too large for that level of effort (my research set was a mere 592 cases of Leukemia). But, detecting and correcting missing data is becoming a whole lot easier due to the tremendous growth in computational power. At work, we spend quite a bit of our time writing tools to detect gaps in data. When they are found, we report it to Corporate Modeling, who then chase after the providers (typically, actuaries in our remote offices) to fill it in. The result is that we have an incredibly robust set of projections data. That allows us to confidently trim the buffer between our actual and required capital and price more competitively. In the context of 4 trillion dollars of insurance in force, the million or so we spend doing this seems like effort well directed.

Terms from Data Mining ch2

More poorly organized notes, primarily definitions from Data Mining.

Data type portability - ease at which a data type can be transformed to another type required for analysis.

Data type porting - the process of transforming. Can be as simple as casting or may involve significant ETL.

Imputation - assigning a best guess for a missing value based on other values in the observation and correlations in the data set.

Latent Semantic Analysis (LSA) - Transforming text to a nonsparse space of lesser dimension through use of word counts and covariances. Leverages the following two properties of text:

• Synonymy - multiple values mean the same thing
• Polysemy - a value can mean more than one thing based on context

Symbolic Aggregate Approximation (SAX) - Method for converting time series to discrete sequence. Aggregates value over time windows and creates a discrete approximation for the entire window.

Discrete Wavelet Transform (DWT) and Discrete Fourier Transform (DFT) - methods for converting time series to series of real-valued coefficients.

Multidimensional Scaling (MDS) - Method for converting graph information to multidimensional vectors for purposes of determining distance between graphs. Typically uses Euclidean measure on resulting vectors, but can use other distance functions as in nonmetric MDS.

Neighborhood graph - used to represent similarity along dimension of any type.

Sampling techiques:

• Biased - intentionally shifting sampling distribution to get more "interesting" data.
• Stratified - form of biased sampling that assures representation in each of several segments of data.
• Reservoir - used to ensure proper representation of moving sample from stream data.

Principal Component Analysis (PCA) - rotating multidimensional basis to reduce number of independent variables (or, more precisely, leveraging correlations in data to capture the information using fewer variables.

Singular Value Decomposition (SVD) - More general case of PCA.

Mean centering - transforming scalar values so the have a mean of zero. PCA and SVD yeild the same results when values are mean centered.

Energy of a transformation - sum of the squared distances from the origin of the original observations projected onto a orthonormalized subspace basis. SVD maximizes energy under this transformation.

Spectral decomposition (also, Schur decomposition) - Diagonalization of a symmetric matrix used by SVD.

Homophily - tendency (preference) of individuals to repeat joint ventures with those with whom they have prior experience. Leads to highly correlated data.