Mastering .NET Machine Learning

Chapter 10. Big Data and IoT

Up to this point, this book has followed a pattern of extracting data, cleaning and shaping the data, and then building machine learning models. A common element in all of the examples is that when we've extracted data, we have brought it from the server (or other external sources) locally to our machine. This means our analysis is confined to whatever data fits in the memory on our local machines. While this is good for small- and medium-sized datasets, there are plenty of datasets and questions that do not fit in RAM. The last couple of years have seen the rise of big data, where we can ask questions of datasets that are too large, unstructured, or fast-moving to be analyzed using our conventional machine learning ...