This one sort of took me by surprise…. Frankly, I’m a little disappointed in myself that I did not think about it or see it as the real need it is.
http://radar.oreilly.com/2014/12/fast-data-calls-for-new-ways-to-manage-its-flow.html
Fast data calls for new ways to manage its flow.
Examples of multi-layer, three-tier data-processing architecture.
Like CPU caches, which tend to be arranged in multiple levels, modern organizations direct their data into different data stores under the principle that a small amount is needed for real-time decisions and the rest for long-range business decisions. This article looks at options for data storage, focusing on one that’s particularly appropriate for the “fast data” of big data.
In other words, IoT is going to generate big data. In what type of storage is that data written to?
Well, it depends.
If you need to crunch the data before you pass it to an AI engine, you might want to buffer the data in some fast memory so you can tweak it for the AI brain.
If you don’t care about it, you just want to keep it, then it can go straight to slow but cheap storage.
If you want to compress it before it’s stored, then you might write it to some slower, but not deep storage speed, memory so it can be compressed by a formula that allows you to get the information back out of it at some future stage.
It makes sense that you would need to consider this, but it totally blew past me. At this stage my house is not generating that much data and I am not storing every little event. I am still at the data collecting stage, the noise stage, so have not had to deal with this topic.
Some more food for future thought on this one will be required.
