book

Designing with Data

by Rochelle King, Elizabeth F Churchill, Caitlin Tan

March 2017

Beginner to intermediate

370 pages

9h 55m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Design and Data: A Perfect SynergyOur Focus: A/B TestingSome Orienting PrinciplesWho Is This Book For?ScopeAbout UsA Word from RochelleA Word from ElizabethA Word from CaitlinHow This Book Is OrganizedHow to Read This BookIntroducing our “Running a Camp” MetaphorO’Reilly SafariHow to Contact UsAcknowledgmentsRochelleElizabethCaitlin
Data as a TrendThree Ways to Think About DataWhat Does This Mean for You as a Designer?Data Can Help to Align Design with BusinessOn Data QualityWith a Little Help from Your Friends...Data ProducersData ConsumersWhat If You Don’t Have Data Friends (Yet)?Themes You’ll See in This BookSummaryQuestions to Ask Yourself
The Diversity of DataMany Dimensions of DataWhy are you collecting data?When is the data collected?How is the data collected?How much data to collect?Why Experiment?Learning About CausalityStatistically Significant, not AnecdotalInformed Opinions about what will happen in the WildBasics of ExperimentationLanguage and ConceptsRace to the Campsite!Experimentation in the Internet AgeA/B Testing: Online ExperimentsSampling Your Users OnlineCohorts and segmentsDemographic informationNew users versus existing usersMetrics: The Dependent Variable of A/B TestingDetecting a Difference in Your GroupsHow big is the difference you want to measure?A big enough sample to power your testSignificance levelYour Hypothesis and Why It MattersDefining a Hypothesis or HypothesesKnow What You Want to LearnRunning Creative A/B TestsData Triangulation: Strength in Mixed MethodsThe Landscape of Design ActivitiesExploring and evaluating IdeasThinking Global and Thinking LocalSummaryQuestions to Ask Yourself
Introducing Our FrameworkWorking with Data Should Feel Familiar...Three Phases: Definition, Execution, and AnalysisThe Definition PhaseThe Execution PhaseThe Analysis PhaseExamples: Data and Design in ActionSummaryQuestions to Ask Yourself
Getting Started: Defining Your GoalDefining Your Metric of InterestMetric sensitivityTracking multiple metricsGetting the full pictureYour metrics may change over timeCompeting metricsRefining Your Goals with DataIdentifying the Problem You Are SolvingRemember Where You AreBuilding Hypotheses for the Problem at HandExample: A Summer Camp HypothesisExample: Netflix—transitioning from DVD rentals to StreamingThe Importance of Going BroadMultiple Ways to Influence a MetricFocus on New and Existing UsersRevisit the Scope of Your ProblemExample: Netflix on the PlayStation 3Involve Your Team and Your Data FriendsWhich Hypotheses to Choose?Consider Potential ImpactUsing What You Already KnowUsing Other Methods to Evaluate Your HypothesesConsider the Reality of Your TestHow much measurable impact do you believe your hypothesis can make?Can you draw all the conclusions you want to draw from your test?Balancing learning and speedKeep Your Old Hypotheses in Your Back PocketSummaryQuestions to Ask Yourself
Designing to LearnEngaging Your Users in a ConversationHaving Quality ConversationsDesigning to extremes to learn about your usersRevisiting the minimum detectable effectDesigning the Best Representation of Your HypothesisUnderstanding Your VariablesNot all variables are visibleYour Design Can Influence Your DataExample: Netflix WiiRevisiting the Space of Design ActivitiesAvoiding Local MaximaDifferent problems for summer campDirectional testing: “Painted door” testsPicking the right level of granularity for your experimentExample: Netflix on Playstation 3Example: Spotify NavigationExperiment 1: Defining the hypothesis to get early directional feedbackExperiment 1: Designing the hypothesesInterlude: Quick explorations using prototypes and usability testingExperiment 2: Refining the “tabbed” navigation“Designing” your testsOther Considerations When Designing to LearnPolishing your design too much, too earlyEdge cases and “worst-case” scenariosTaking advantage of other opportunities to learn about your designIdentifying the Right Level of Testing for Different Stages of ExperimentationRunning parallel experimentsThinking about “Experiment 0”SummaryQuestions to Ask Yourself
Vetting Your Designs Ahead of LaunchLab Studies: Interviews and Usability TestingSurveysWorking with Your Peers in DataLaunching Your DesignBalancing Trade Offs to Power Your TestWeighing sample size and significance levelGetting the sample that you need (rollout % versus test time)Who are you including in your sample?Practical Implementation DetailsIs your experience “normal” right now?Sanity check: Questions to ask yourselfEvaluating Your ResultsRevisiting Statistical SignificanceWhat Does the Data Say?Expected (“Positive”) ResultsUnexpected and Undesirable (“Negative”) ResultsWhen the World is FlatErrorsReplicationUsing secondary metricsUsing multiple test cellsRolling out to more usersRevisiting “thick” dataGetting Trustworthy DataNovelty effectSeasonality biasRolling Out Your Experience, or NotWhat’s Next for Your Designs?Were you exploring or evaluating?Was your problem global or local?Knowing when to stopRamp UpHoldback GroupsTaking Communication into AccountCase Study: Netflix on PlayStation 3Many Treatments of the Four HypothesesEvolving the Design Through Iterative TestsWhat If You Still Believe?SummaryQuestions to Ask Yourself
Principle 1: Shared Company Culture and ValuesDepth: Communicating Across LevelsBreadth: Beyond Design and ProductThe Importance of a Learning CultureThe rewards of taking risks: Redefining “failure”The value of developing your customer instinctPrinciple 2: Hiring and Growing the Right PeopleEstablishing a Data-Aware Environment Through Your PeersHiring for SuccessBuilding the team with data involved from the startPrinciple 3: Processes to Support and AlignEstablishing a Knowledge BaselineEstablishing a Common VocabularyDeveloping a Rhythm Around Data Collection and SharingProject review meetingsSpreading data across the organizationCreating a Presence in the OfficeLearning from the PastSummaryQuestions to Ask Yourself

Ethical ConsiderationsEthics in Online ExperimentationDesign Experimentation Versus Social ExperimentationTwo “Power of Suggestion” ExperimentsToward Ethical A/B TestingKey ConceptsAsking Questions, Thinking EthicallyLast Words
KeywordsChapter 1Chapter 2Chapter 3Chapters 4, 5, and 6Chapter 7Chapter 8BooksOnline Articles, Papers, and BlogsCoursesToolsProfessional Groups, Meetups, and Societies

Content preview from Designing with Data

Chapter 8. Conclusion

WE HAVE COVERED A lot of ground in this book. We hope you have enjoyed reading it as much as we have enjoyed writing it.

Throughout the book we have been arguing that designers will gain value from, and will also add immense value to, the design of A/B tests. Hopefully, you now understand the basics of A/B testing, how A/B testing enables you to compare one or more versions of a designed service experience, and how A/B testing can help you understand more about your users’ needs. The benefits of conducting A/B experiments are being able to get access to a large number of users at one time in their real-world context, and the ability to compare many different versions of your design(s) simultaneously. A/B tests can improve product performance for different cohorts of users, and can help you figure out how to attract and retain new users. As the purpose of A/B testing is to evaluate which experiences perform better for users and for your business according to previously defined, relevant metrics (such as which links or buttons are clicked or how long someone stays on a site), we hope that you are now also intrigued by understanding what metrics work best for your business context.

Beyond specific features, products, or business models, though, A/B tests can help you develop an understanding of human behavior more generally; they can help you to hone your skills regarding what works best for your users in your interfaces and in your information and service design. ...