Although the orders stream doesn’t currently contain detailed information about products, in the last chapter we set up Debezium to capture any changes to the MySQL products table and write them into the products stream.
In this chapter, you’ll learn how to combine the orders stream and products streams by using a stream processor.
We’ll put the new stream into Apache Pinot and update the dashboard with the top-selling products and categories.
Enriching Orders with Kafka Streams
In Chapter 4, we used Kafka Streams to create a windowed aggregation over the orders stream so that we could compute the number of orders and revenue in the last few minutes.
In this section, we’re going to populate a new stream called enriched-order-items that will contain all the order items contained in the orders stream, hydrated with details from the products stream.
Figure 8-1 shows what we’re going to build in more detail.
Figure 8-1. Architecture with enriched orders using Kafka Streams
We can break Figure 8-1 down further to visualize the different processors that will exist in the Kafka Streams graph to achieve this, as shown in Figure 8-2.
Figure 8-2. Kafka Streams processors
Now that we have an overview of what we’re going to build, let’s get ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.
