Streaming Databases

Chapter 11. Future State of Real-Time Data

The cave you fear to enter holds the treasure you seek.

Joseph Campbell

After delving deeply into the deployment options for streaming databases, this chapter takes a step back and looks into the future state of real-time data, shaped by one of the central topics of this book: the accelerated convergence of streaming and databases. Streaming databases are one of the manifestations of this trend. But there’s so much more evidence worth at least touching upon here.

We start out with graph databases and their ongoing journey toward the streaming realm (e.g., Memgraph, thatDot), followed by nowadays, after the ChatGPT GenAI breakthrough, super-popular vector databases (e.g., Milvus, Weaviate). We continue our travels through the converging lands of streaming and databases with tools for bringing one central aspect of streaming databases, namely, Materialized Views, aka IVM, to classical databases (Feldera, PeerDB, and Epsio). Toward the end of this chapter, we examine the streaming functionalities of established database vendors such as MongoDB and slowly turn our focus to the analytical plane with data warehouses like BigQuery, Redshift, and Snowflake that are also consequently extending their streaming functionalities. We close this chapter by surveying the confluence of streaming and lakehouse architectures driven by Apache Iceberg, Apache Hudi, Delta Lake, and Apache Paimon—one of the most promising macro trends not only in the streaming ...