Software Stream of Consciousness: The End of the Line for the Business Intelligence ...

Software Stream of Consciousness: The End of the Line for the Business Intelligence Cubes ...: "I outline a plan on how Bayesian Statistics Analysis can be incorporated into a database engine to digest a tsunami of data in real time and not only make sense of it, but use it to predict things." Read More.

Digital Data DNA

We are generating data at an incredible rate. With the help of Moore's Law which states that the amount of transistors (hence) memory on a chip doubling every two years, we now take pictures, record and twitter every moment of our lives. All of this digital data has to be stored. Up until now it hasn't been a problem.

However, the most stuff that we as humans are generating is data and the curve is not linear but exponential. This means that the amount of yearly data is not a straight line but a power curve. If we generate 2 exabytes a year, then the next year it is 4 and the next year is 16 exabytes. We must find new ways to manage this data.

Mother Nature has given us a template in DNA. DNA contains enough encoded information to make a human being, and it is tiny and found in every cell. We must find a way to encode data in a manner similar to the way DNA encodes information. Right now, data is a single dimension. If we can dimensionalize data, we can store a lot more data in a much smaller space. It will be much more efficient than the data compression that we do now.

So all of you inventors out there, get your thinking caps on. If you find a way to make Digital Data DNA, you will be richer than Bill Gates, Warren Buffett, Carlos Slim and the top 5 worlds richest billionaires all combined.

Data Wrappers

One of the biggest problems with data is that it is quite useless without an application to do something with it. When we get a file, we rely on it's file extension to tell us what it is. For example, when we see .jpg, we know that we can expect to see a picture.

This is quite inefficient. For example, data on a Mac is usually not transferable to data on a PC unless we have appropriate programs on both ends.

What the world needs is an innovative new data structure. This would be a wrapper on the outside of sort of a universal envelope. The envelope will tell us what is inside, who sent it, who created it, when it was last modified, and it self-executes. This means that it can display itself on any computer without a program.

The wrapper could be so configured that it would be impossible to hide a virus inside the data. And when you pop this data into a database, it is self-sorting because it knows about itself.

New data handling methods are going to be required, because our ability to generate data is outstripping our ability to handle it. The only thing growing faster than the amount of data we generate, is the amount of bullshit that politicians spew. And that isn't worth storing.