Distributed Systems with Node.js

Chapter 9. Distributed Primitives

Data primitives are rather straightforward when dealing with a single-threaded program. Want to make a lock? Just use a boolean. Want a key/value store? A Map instance is your friend. Want to keep an ordered list of data? Reach for an array. When only a single thread reads and writes to an array, it’s as simple as calling Array#push() and Array#pop(). In this situation, the array instance is the complete source of truth. There are no other copies that can get out of sync, no messages in transit that can be received out of order. Persisting the data to disk is as easy as calling JSON.stringify() and fs.writeFileSync().

Unfortunately, the performance impact of such an approach is huge, and scaling to a sizeable userbase is nearly impossible. Not to mention such a system has a single point of failure! Instead, as you’ve seen throughout this book, the answer to performance and avoiding a single point of failure depends on redundant distributed processes. Care must be put into the storage and manipulation of data, particularly when it comes to distributed systems.

Not every problem can be solved using the same data store. Depending on the data requirements—such as entity relationships, the amount of data, and requirements with consistency, durability, and latency—different solutions must be chosen. It’s not uncommon for an application composed of distributed services to require several data storage tools. Sometimes you need a graph database and sometimes ...