Building Real-Time Data Pipelines

Chapter 1. When to Use In-Memory Database Management Systems (IMDBMS)

In-memory computing, and variations of in-memory databases, have been around for some time. But only in the last couple of years has the technology advanced and the cost of memory declined enough that in-memory computing has become cost effective for many enterprises. Major research firms like Gartner have taken notice and have started to focus on broadly applicable use cases for in-memory databases, such as Hybrid Transactional/Analytical Processing (HTAP for short).

HTAP represents a new and unique way of architecting data pipelines. In this chapter we will explore how in-memory database solutions can improve operational and analytic computing through HTAP, and what use cases may be best suited to that architecture.

Improving Traditional Workloads with In-Memory Databases

There are two primary categories of database workloads that can suffer from delayed access to data. In-memory databases can help in both cases.

Online Transaction Processing (OLTP)

OLTP workloads are characterized by a high volume of low-latency operations that touch relatively few records. OLTP performance is bottlenecked by random data access—how quickly the system finds a given record and performs the desired operation. Conventional databases can capture moderate transaction levels, but trying to query the data simultaneously is nearly impossible. That has led to a range of separate systems focusing on analytics more than transactions. ...