Operating Continuously

Chapter 4. Measure and Experiment

Up to this point our model has consisted of deploy, release, and operate. For the final addition to our continuous operations model, we need to measure the impacts of our changes through data collection and run experiments against that data to validate whether those changes yielded the results we expected or desired. A robust measurement and experimentation practice yields treasure troves of data that can inform better product direction, reduce the risk of degradations in service, or even give an early warning sign of misaligned feature development.

In Chapter 3, we briefly touched on monitoring and observability solutions. We discussed that application monitoring tools often rely on reactive alerting techniques to inform operators about changes to application and platform performance. In contrast, the measure and experiment stage focuses on quantifying the efficacy of the features you are releasing to end users more proactively. As teams progress along their continuous operations and feature management journey, understanding the metrics that support a successful feature release—and, in some cases, support rolling back a feature—becomes increasingly important. These measurements have a direct relationship to the features you are releasing separately from your software deployment. It’s necessary to shift measure and experiment practices closer to the release stage (see Figure 4-1).

Figure 4-1. The measure and experiment stage

But why is measure ...