Designing with Data

Chapter 6. The Analysis Phase (Getting Answers From Your Experiments)

LAUNCHING YOUR A/B TEST out “in the wild” is exciting—it’s the first time you’ll get to have a conversation with your users at scale about how your designs will serve them and meet their needs! This process of launching and learning from your test will be our focus in this chapter. One of the principles behind A/B testing is that it’s a good idea to always test first with a very small portion of your user base. The data you get back from your experiments will give you a sense of how well your design is performing with respect to your goals, before you invest the time to launch a design to all of your users. Did the design(s) lead to the behaviors you were expecting to see? Did you see that people acted in ways which map more to your business metrics? Were you able to successfully improve the metrics that you were targeting? For example, if your goal is that your users watch more movies they like, did you see an uptick in movies selected and watched all the way through? This would indicate that your recommendation algorithms and/or your presentation of selections are more effective than in your previous design.

Once you get back data from your initial tests, you then need to decide what you’ll do next. Do you take that experience and then roll it out to a larger part of your user base? Did the results suggest you are on the wrong track, that you should abandon the idea entirely? Or do you keep working on your hypothesis ...