97 Things Every Data Engineer Should Know

Chapter 89. What Is Big Data?

Ami Levin

You’ve heard the term big data numerous times.¹ Articles and books have been written about it. You might be using a product that claims to be such. Perhaps you even have it on your resume. But have you ever stopped to think about what it really means?

Big data has no standard, clear, agreed-upon definition. Some use it to describe data volumes, others as an indicator of data velocity, or of data of a large variety. None of these have any quantitative metrics that can classify a dataset as being big or small. Many use it to refer to specific technologies such as Hadoop, while others use it to describe data from specific sources such as social media or IoT devices, or so called unstructured data.

So many conflicting, vague, and ambiguous definitions for big data exist, but none clearly describe the data or its use. Anyone can claim their products, services, technologies, or datasets to be “big data,” and this claim can’t be refuted.

The truth is that there is no such thing as big data. Large-volume, high-speed, varying-source data has always challenged our ability to derive value from it. Nothing fundamental has changed in the last decade to warrant a new term.

Data management has always been about applying analytical methods to gain insights that improve decision quality. Nothing more, nothing less. By now, you may be asking yourself why ...