Designing Data-Intensive Applications, 2nd Edition

Chapter 13. A Philosophy of Streaming Systems

If a thing be ordained to another as to its end, its last end cannot consist in the preservation of its being. Hence a captain does not intend as a last end, the preservation of the ship entrusted to him, since a ship is ordained to something else as its end, viz. to navigation.

(Often quoted as: If the highest aim of a captain was the preserve his ship, he would keep it in port forever.)

St. Thomas Aquinas, Summa Theologica (1265–1274)

In Chapter 2 we discussed the goal of creating applications and systems that are reliable, scalable, and maintainable. These themes have run through all the chapters—for example, we discussed many fault-tolerance algorithms that help improve reliability, sharding to improve scalability, and mechanisms for evolution and abstraction that improve maintainability.

In this chapter we will bring all these ideas together and build on the streaming/event-driven architecture ideas from Chapter 12 in particular to develop a philosophy of application development that meets those goals. This chapter is more opinionated than previous chapters, presenting a deep dive into one particular philosophy rather than comparing multiple approaches.

Data Integration

A recurring theme in this book has been that for any given problem, there are several solutions, each with its own pros, cons, and trade-offs. For example, when discussing storage engines in Chapter 4, we examined log-structured storage, B-trees, and column-oriented ...