Building Real-Time Data Pipelines

Chapter 2. First Principles of Modern In-Memory Databases

Our technological race to the future with billions of mobile phones, an endless stream of online applications, and everything connected to the Internet has rendered a new set of modern workloads. Our ability to handle these new data streams relies on having the tools to handle large volumes of data quickly across a variety of data types. In-memory databases are key to meeting that need.

The Need for a New Approach

Traditional data processing infrastructures, particularly the databases that serve as a foundation for applications, were not designed for today’s mobile, streaming, and online world. Conventional databases were designed around slow mechanical disk drives that cannot keep up with modern workloads. Conventional databases were also designed as monolithic architectures, making them hard to scale, and forcing customers into expensive and proprietary hardware purchases.

A new class of in-memory solutions provides an antidote to legacy approaches, delivering peak performance as well as capabilities to enhance existing and support new applications.

For consumers, this might mean seeing and exchanging updates with hundreds or thousands of friends simultaneously. For business users, it might mean crunching through real-time and historical data simultaneously to derive insight on critical business decisions.

Architectural Principles of Modern In-Memory Databases

To tackle today’s workloads and anticipate the needs of ...