Observability Engineering, 2nd Edition

Chapter 15. Cheap and Accurate Enough Sampling

In the preceding two chapters, we examined datastore configurations and implementations that can efficiently store and retrieve large quantities of observability data. In this chapter, we’ll look at techniques for reducing the amount of observability data you may need to store.

At a large enough scale, the resources necessary to retain and process every single event can become prohibitive and impractical. Sampling events can mitigate the trade-offs between resource consumption and data fidelity. This chapter examines why sampling is useful (even at a smaller scale), the various strategies typically used to sample data, and trade-offs between those strategies.

In this chapter, we use code-based examples to illustrate how these strategies are implemented, and progressively introduce concepts that build upon previous examples. We’ll start with simpler sampling schemes applied to single events as a conceptual introduction to using a statistical representation of data when sampling. Then, we build toward more complex sampling strategies as they are applied to a series of related events (trace spans), and propagate the information needed to reconstruct your data after sampling.

Sampling to Refine Your Data Collection

Past a certain point, the cost to collect, process, and save every log entry, every event, and every trace that your systems generate dramatically outweighs the benefits. At a large enough scale, it is simply not feasible ...