Reactive Microsystems

Chapter 6. Toward Scalable Persistence

Anything you build on a large scale or with intense passion invites chaos.

Francis Ford Coppola

Up to this point, we have discussed the evolution of a monolith, through a system of microliths, to a design based on scalable and resilient microsystems. But, we have made it too easy for ourselves by ignoring the most difficult problem of them all: scaling state and, in particular, durable state.

Let’s address this and look at how we can scale state (while staying available) in an event-based design, alongside the new opportunities and constraints such a design enables. First and foremost, it requires a new way of thinking about persistence and consistency, with the first step being to move beyond CRUD.

Moving Beyond CRUD

When bookkeeping was done with clay tablets or paper and ink, accountants developed some clear rules about good accounting practices.

One never alters the books; if an error is made, it is annotated and a new compensating entry is made in the books. The books are thus a complete history of the transactions of the business.

Update-in-place strikes many systems designers as a cardinal sin: it violates traditional accounting practices that have been observed for hundreds of years.

Jim Gray, The Transaction Concept, 1981

Disk space used to be very expensive. This is one of the reasons why most SQL databases are using update-in-place—overwriting existing records with new data as it arrives.

As Jim Gray, Turing Award winner and ...