Streaming Databases

Chapter 1. Streaming Foundations

The hero’s journey always begins with the call. One way or another, a guide must come to say, “Look, you’re in Sleepy Land. Wake. Come on a trip. There is a whole aspect of your consciousness, your being, that’s not been touched. So you’re at home here? Well, there’s not enough of you there.” And so it starts.

Joseph Campbell, Reflections on the Art of Living: A Joseph Campbell Companion

The streaming database is a concept born from over a decade of processing and serving data. The evolution leading to the advent of streaming databases is rooted in the broader history of database management systems, data processing, and the changing demands of the digital age. To understand this evolution, let’s take a historical journey through the key milestones that have shaped the development of streaming databases.

The rise of the internet and the explosive growth of digital data in the late 20th century led to the need for more scalable and flexible data management solutions. Data warehouses and batch-oriented processing frameworks like Hadoop emerged to address these challenges of the size of data during this era.

The term “big data” was and still is used to refer not only to the size of data but also to all solutions that store and process data that is extremely large. Big data cannot fit on a single computer or server. You need to divide it up into smaller, equal-sized parts and store them in multiple computers. Systems like Hadoop and MapReduce became ...