Mastering Distributed Tracing

Partial sampling

Let's conclude the overview of the sampling techniques with an approach used in some tracing systems where sampling decision does not guarantee consistent collection of all spans of a trace. It does not mean the sampling decision is made completely randomly at every node of the call graph, but rather that only a portion of the call graph is sampled. Specifically, the sampling decision can be made after detecting an anomaly in the trace, such as unusual latency or an error code. The tracing library is changed slightly to keep all the spans for currently active traces in memory until the entry span is finished. Once the sampling decision is made, we send all those spans to the tracing backend. Even though we will miss the spans ...