Observability Engineering, 2nd Edition

Chapter 6. Making Structured Events Arbitrarily Wide

This chapter is contributed by Jeremy Morrell, Principal Engineer at Cloudflare.

A Note from Charity, Liz, George, and Austin

In the last chapter, we broke down how metrics, logs, and traces can all be decomposed into one common datatype: the arbitrarily wide, structured event—which can be used to generate any view necessary during observability-based investigations. However, we mostly focused on showing examples that illustrate the “structured event” part of that datatype. In this chapter, we show you how to make those structured events “arbitrarily wide.”

In observability, your structured events should be as wide as possible in order to capture comprehensive context for debugging. Having rich context about any given system event enables you to answer unexpected questions and perform complex analysis without prejoining data. Some of the most useful data when investigating novel or complex failures will typically involve comparing outliers across fields with high cardinality.

Wide, structured events collapse the boundaries between logs, metrics, and traces into a single data structure, offering a unified view for analyzing system behavior. That means proactively designing your events so wide that they include fields to help pinpoint issues such as user ID, feature flags, and request details in a single data structure, eliminating the need for expensive and complex joins.

An “arbitrarily wide” structured event means that there ...