Growth Engineering

CHAPTER 7Introduction to A/B Testing

In the previous chapters, we explored what experiments are and the different ways teams use them to learn. Among all the forms of experimentation, one stands out for its simplicity and power: A/B testing. It has become the common language of product growth, used by startups testing a new onboarding flow, by e-commerce companies refining pricing, and by enterprise teams optimizing feature adoption.

At its core, an A/B test is deceptively simple. One version is shown to a group of users, another version to a different group, and the outcomes are compared. Yet behind that simplicity lies deep statistical thinking. Every decision about who sees what, how long the test runs, and when to call a winner is grounded in probability and inference. Without those foundations, an A/B test risks becoming an illusion of evidence rather than a source of truth.

This chapter unpacks the science behind that simplicity. We will use A/B testing as a lens to understand the statistical concepts that make experimentation trustworthy, including significance, power, variance, and error. You will also learn how to design and interpret tests correctly so that what you measure truly reflects user behavior and business impact.

Growth is not driven by how many tests you run but by how many truths you uncover. When done well, A/B testing helps teams find those truths with clarity and discipline.

A/B testing may seem simple on the surface, but running one well requires discipline. ...