Unlike the other tests that we've seen so far, A/B tests do not rely on a unique test statistic and a unique distribution to derive inference; as a matter of fact, they could benefit from any test statistic and distribution. A/B testing is, rather, a name given to a broad technique used for versions comparison that will dictate things from how to sample all of the way to getting your p-value and confidence interval and making a decision. These tests are wildly popular in the field of web analytics, for very good reasons, but are not restricted to it.
These tests are capable of handling statistically based insights to a broad set of is A better than B? kind of problems. Will ...
