Building Machine Learning Systems with Python - Third Edition

Bigger Data

It's not easy to say what big data is. We will adopt an operational definition: when data is so large that it becomes cumbersome to work with, we refer to it as big data. In some cases, this might mean petabytes of data or trillions of transactions: data that will not fit into a single hard drive. In other cases, it may be one hundred times smaller, but still difficult to work with.

Why has data itself become an issue? While computers keep getting faster and gaining more memory, the size of the data has grown as well. In fact, data has grown faster than computational speed and few algorithms scale linearly with the size of the input data taken together; this means that data has grown faster than our ability to process it.

We will ...