97 Things Every Data Engineer Should Know

Chapter 33. Friends Don’t Let Friends Do Dual-Writes

Gunnar Morling

Long gone are the times when enterprise application developers could just put data into their application’s database and call it a day. Nowadays, data often needs to be persisted in multiple systems in order to satisfy ever-increasing user requirements: changed data needs to be sent to separate search services, enabling a feature-rich full-text search experience usually not provided by databases themselves, and caches need to be updated, allowing for fast data retrieval without accessing the database itself.

But how can all these data stores—an application’s database, a search index, a cache—be kept in sync? We might be tempted to simply issue requests to all these systems for updating the data, but here be dragons.

What if, for instance, the search index temporarily isn’t available because of a networking issue? We could think of implementing retry logic, but things quickly get complex there. Worse, even if we are able to successfully update all the involved resources, we might end up with data inconsistencies. Unless all the updates are done within a single global transaction (which, in turn, would tie our service quality to the availability of all the resources), we have no ordering guarantees across multiple data changes. If two updates to the same record occur in close temporal proximity, they could be applied ...