2 A/B testing: Evaluating a modification to your system
This chapter covers
- Randomizing to remove measurement bias
- Replicating to reduce measurement variation
- Determining how many measurements to take
- Deciding whether to accept or reject your system change
In chapter 1, you saw that the final step in the engineer’s workflow is to measure how business metrics are impacted by a modification of your system. You do this by running an experiment on the modified production system. Experiments are the most accurate way to measure changes in business metrics.
In this chapter, you’ll learn how to run an A/B test, the simplest and most widely used type of experiment. An A/B test measures the business metric for each of version A and B. If you find that ...
Get Experimentation for Engineers now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
