Scaling Search and Retrieval for Contextual AI

Chapter 7. Sharding and Index Partitioning

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 7th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Objective: Learn how to scale search horizontally using shard design, routing strategies, and segment-aware partitioning.

Up to this point, the analysis of search and retrieval systems has relied on a critical but implicit assumption: the index exists as a single logical unit. Even as concurrency, durability, and performance tradeoffs in the write path were explored, the discussion assumed a unified segment space governed by a single lifecycle. This assumption is intentional. A system must first achieve correctness, performance, and durability in isolation before it can be scaled outward.

Chapter 7 intentionally challenges this foundational assumption.

Horizontal scaling does not introduce a new indexing model, replace segments, alter scoring logic, or discard previously established durability boundaries. Instead, it raises a more nuanced and challenging question: how do the guarantees defined for a single index persist when the index is partitioned ...