Mastering Kafka Streams and ksqlDB

Chapter 12. Testing, Monitoring, and Deployment

In the previous chapters, we learned how to build a variety of stream processing applications with Kafka Streams and ksqlDB. In this final chapter, we will learn some steps we’ll need to take in order to ship our applications to a production environment. You may be wondering why we decided to unify the chapter on productionizing Kafka Streams and ksqlDB applications. Despite some differences, specifically with respect to testing, the process is largely the same, and simplifying our mental model for how we productionize software will improve its maintainability in the long run (especially in hybrid environments where we use both Kafka Streams and ksqlDB).

Some of the questions we will answer in this chapter include:

• How can we test Kafka Streams applications and ksqlDB queries?

• How can we run benchmarks against a Kafka Streams topology?

• Which types of monitoring should we have in place?

• What is the process for accessing the built-in JMX metrics in Kafka Streams and ksqlDB?

• How can we containerize and deploy our Kafka Streams and ksqlDB applications?

• What are some of the operational tasks we are likely to encounter?

Let’s start by learning how to test our stream processing applications.

Testing

Once the initial development of your ksqlDB queries or Kafka Streams application is complete, you will likely continue shipping updates to your code over time. For example, a change in business requirements may require an update ...