AATD has come to the conclusion that it’s going to need to introduce a new piece of infrastructure to achieve scalable real-time analytics, but isn’t yet convinced that a full-blown OLAP database is necessary.
In this chapter, we’ll start by explaining why we can’t just use a stream processor to serve queries on streams, before introducing Apache Pinot, one of the new breed of OLAP databases designed for real-time analytics.
We’ll learn about Pinot’s architecture and data model, before ingesting the orders stream.
After that, we’ll learn about timestamp indexes and how to write queries against Pinot using SQL.
Figure 5-1 shows how we’re going to evolve our infrastructure in this chapter.
Figure 5-1. Evolution of the orders service
Why Can’t We Use Another Stream Processor?
At the end of the last chapter, we described some of the limitations of using Kafka Streams to serve queries on top of streams. (See “Limitations of Kafka Streams”.)
These were by no means a criticism of Kafka Streams as a technology; it’s just that we weren’t really using it for the types of problems for which it was designed.
A reasonable question might be, Why can’t we use another stream processor instead, such as ksqlDB or Flink?
Both of these tools offer SQL interfaces, solving the issue of having to write Java code to query streams.
Unfortunately, it still doesn’t ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
Julian F.
Head of Cybersecurity
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.
Addison B.
Field Engineer
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.
Amir M.
Data Platform Tech Lead
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
