Designing Data-Intensive Applications, 2nd Edition

Chapter 7. Sharding

Clearly, we must break away from the sequential and not limit the computers. We must state definitions and provide for priorities and descriptions of data. We must state relationships, not procedures.

Grace Murray Hopper, Management and the Computer of the Future (1962)

A distributed database typically distributes data across nodes in two ways:

• It stores a copy of the same data on multiple nodes. This is replication, which we discussed in Chapter 6.

• If there’s so much data or such a high write throughput that a single node cannot handle it, it splits the data into smaller shards or partitions, and stores different shards on different nodes. We’ll discuss sharding in this chapter.

Normally, shards are defined in such a way that each piece of data (each record, row, or document) belongs to exactly one shard. There are various ways of achieving this, which we will discuss in depth in this chapter. In effect, each shard is a small database of its own, although some database systems support operations that touch multiple shards at the same time.

Sharding is usually combined with replication, so that copies of each shard are stored on multiple nodes. This means that even though each record belongs to exactly one shard, it may still be stored on several different nodes for fault tolerance.

A node may store more than one shard. If a single-leader replication model is used, the combination of sharding and replication can look like Figure 7-1, for example. ...