The Self-Service Data Roadmap

Chapter 14. A/B Testing Service

Now we are ready to operationalize our data and ML pipelines to generate insights in production. There are multiple ways to generate the insight, and data users have to make a choice about which one to deploy in production. Consider the example of an ML model that forecasts home prices for end customers. Assume there are two equally accurate models developed for this insight—which one is better? This chapter focuses on an increasingly growing practice where multiple models are deployed and presented to different sets of customers. Based on behavioral data of customer usage, the goal is to select a better model. A/B testing (also known as bucket testing, split testing, or controlled experiment) is becoming a standard approach for evaluating user satisfaction from a product change, a new feature, or any hypothesis related to product growth. A/B testing is becoming a norm, and is widely used to make data-driven decisions. It is critical to integrate A/B testing as a part of the data platform to ensure consistent metrics definitions are applied across ML models, business reporting, and experimentation. While A/B testing could fill a complex, full-fledged book by itself, this chapter covers the core patterns in the context of the data platform as a starting point for data users.

Online controlled A/B testing is utilized at a wide range of companies to make data-driven decisions. As noted by Kohavi and Thomke, A/B testing is used for anything from frontend ...