Designing Data-Intensive Applications, 2nd Edition

Chapter 7. Sharding

Clearly, we must break away from the sequential and not limit the computers. We must state definitions and provide for priorities and descriptions of data. We must state relationships, not procedures.

Grace Murray Hopper, Management and the Computer of the Future (1962)

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 7th chapter of the final book. The GitHub repo for this book is https://github.com/ept/ddia2-feedback.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out on GitHub.

A distributed database typically distributes data across nodes in two ways:

1. Having a copy of the same data on multiple nodes: this is replication, which we discussed in Chapter 6.

2. If we don’t want every node to store all the data, we can split up a large amount of data into smaller shards or partitions, and store different shards on different nodes. We’ll discuss sharding in this chapter.

Normally, shards are defined in such a way that each piece of data (each record, row, or document) belongs to exactly one shard. There are various ways of achieving this, which we discuss in depth in this chapter. In effect, each shard is a small database of its own, although some database systems support operations that touch multiple ...