Scaling Search and Retrieval for Contextual AI

Chapter 4. Segment Files and Storage Models

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 4th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Search engines rely on far more than clever indexing to achieve real-world performance. Beneath every fast query response is a carefully engineered storage layer. As illustrated in Figure 4-1, this layer should balance write throughput, read latency, durability, and scale. Unlike traditional analytics engines, which favor columnar formats optimized for large scans ideal for aggregation use cases, search engines require data structures that support low-latency random access, rapid document filtering, and complex scoring. These capabilities must also juggle frequent writes, updates, and deletions.

Figure 4-1. Segment files are self-contained, immutable indexes.

This chapter covers how modern search engines persist indexed content to disk, and how that layout enables (or limits) future retrieval. We explore the concept of a segment file as the core ...