Thursday, September 24, 2015

Lots of data vs. Big Data

We use huge databases at work. The primary one I support is 15Tb and gets fed from one that's nearly 100Tb. But, while it certainly satisfies the first two "v's" of Big Data (volume and velocity), it doesn't really satisfy the third, variety. Our data is highly structured and homogeneous. That's why we've been able to keep it on relational platforms this long. The day has come, however, to start seriously looking at other architectures. We see year over year growth in the range of 50% which means even my "small" database will be up around 25Tb by the end of 2016. We're already seeing significant performance degradation due to volume. So, I got to spend most of this week talking with some experts in the Big Data space. Lot's of good stuff and quite relevant to my research direction. From a first blush, these architectures actually support sampling far better than traditional relational models.

No comments:

Post a Comment