book

Designing with Data

by Rochelle King, Elizabeth F Churchill, Caitlin Tan

March 2017

Beginner to intermediate

370 pages

9h 55m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Design and Data: A Perfect SynergyOur Focus: A/B TestingSome Orienting PrinciplesWho Is This Book For?ScopeAbout UsA Word from RochelleA Word from ElizabethA Word from CaitlinHow This Book Is OrganizedHow to Read This BookIntroducing our “Running a Camp” MetaphorO’Reilly SafariHow to Contact UsAcknowledgmentsRochelleElizabethCaitlin
Data as a TrendThree Ways to Think About DataWhat Does This Mean for You as a Designer?Data Can Help to Align Design with BusinessOn Data QualityWith a Little Help from Your Friends...Data ProducersData ConsumersWhat If You Don’t Have Data Friends (Yet)?Themes You’ll See in This BookSummaryQuestions to Ask Yourself
The Diversity of DataMany Dimensions of DataWhy are you collecting data?When is the data collected?How is the data collected?How much data to collect?Why Experiment?Learning About CausalityStatistically Significant, not AnecdotalInformed Opinions about what will happen in the WildBasics of ExperimentationLanguage and ConceptsRace to the Campsite!Experimentation in the Internet AgeA/B Testing: Online ExperimentsSampling Your Users OnlineCohorts and segmentsDemographic informationNew users versus existing usersMetrics: The Dependent Variable of A/B TestingDetecting a Difference in Your GroupsHow big is the difference you want to measure?A big enough sample to power your testSignificance levelYour Hypothesis and Why It MattersDefining a Hypothesis or HypothesesKnow What You Want to LearnRunning Creative A/B TestsData Triangulation: Strength in Mixed MethodsThe Landscape of Design ActivitiesExploring and evaluating IdeasThinking Global and Thinking LocalSummaryQuestions to Ask Yourself
Introducing Our FrameworkWorking with Data Should Feel Familiar...Three Phases: Definition, Execution, and AnalysisThe Definition PhaseThe Execution PhaseThe Analysis PhaseExamples: Data and Design in ActionSummaryQuestions to Ask Yourself
Getting Started: Defining Your GoalDefining Your Metric of InterestMetric sensitivityTracking multiple metricsGetting the full pictureYour metrics may change over timeCompeting metricsRefining Your Goals with DataIdentifying the Problem You Are SolvingRemember Where You AreBuilding Hypotheses for the Problem at HandExample: A Summer Camp HypothesisExample: Netflix—transitioning from DVD rentals to StreamingThe Importance of Going BroadMultiple Ways to Influence a MetricFocus on New and Existing UsersRevisit the Scope of Your ProblemExample: Netflix on the PlayStation 3Involve Your Team and Your Data FriendsWhich Hypotheses to Choose?Consider Potential ImpactUsing What You Already KnowUsing Other Methods to Evaluate Your HypothesesConsider the Reality of Your TestHow much measurable impact do you believe your hypothesis can make?Can you draw all the conclusions you want to draw from your test?Balancing learning and speedKeep Your Old Hypotheses in Your Back PocketSummaryQuestions to Ask Yourself
Designing to LearnEngaging Your Users in a ConversationHaving Quality ConversationsDesigning to extremes to learn about your usersRevisiting the minimum detectable effectDesigning the Best Representation of Your HypothesisUnderstanding Your VariablesNot all variables are visibleYour Design Can Influence Your DataExample: Netflix WiiRevisiting the Space of Design ActivitiesAvoiding Local MaximaDifferent problems for summer campDirectional testing: “Painted door” testsPicking the right level of granularity for your experimentExample: Netflix on Playstation 3Example: Spotify NavigationExperiment 1: Defining the hypothesis to get early directional feedbackExperiment 1: Designing the hypothesesInterlude: Quick explorations using prototypes and usability testingExperiment 2: Refining the “tabbed” navigation“Designing” your testsOther Considerations When Designing to LearnPolishing your design too much, too earlyEdge cases and “worst-case” scenariosTaking advantage of other opportunities to learn about your designIdentifying the Right Level of Testing for Different Stages of ExperimentationRunning parallel experimentsThinking about “Experiment 0”SummaryQuestions to Ask Yourself
Vetting Your Designs Ahead of LaunchLab Studies: Interviews and Usability TestingSurveysWorking with Your Peers in DataLaunching Your DesignBalancing Trade Offs to Power Your TestWeighing sample size and significance levelGetting the sample that you need (rollout % versus test time)Who are you including in your sample?Practical Implementation DetailsIs your experience “normal” right now?Sanity check: Questions to ask yourselfEvaluating Your ResultsRevisiting Statistical SignificanceWhat Does the Data Say?Expected (“Positive”) ResultsUnexpected and Undesirable (“Negative”) ResultsWhen the World is FlatErrorsReplicationUsing secondary metricsUsing multiple test cellsRolling out to more usersRevisiting “thick” dataGetting Trustworthy DataNovelty effectSeasonality biasRolling Out Your Experience, or NotWhat’s Next for Your Designs?Were you exploring or evaluating?Was your problem global or local?Knowing when to stopRamp UpHoldback GroupsTaking Communication into AccountCase Study: Netflix on PlayStation 3Many Treatments of the Four HypothesesEvolving the Design Through Iterative TestsWhat If You Still Believe?SummaryQuestions to Ask Yourself
Principle 1: Shared Company Culture and ValuesDepth: Communicating Across LevelsBreadth: Beyond Design and ProductThe Importance of a Learning CultureThe rewards of taking risks: Redefining “failure”The value of developing your customer instinctPrinciple 2: Hiring and Growing the Right PeopleEstablishing a Data-Aware Environment Through Your PeersHiring for SuccessBuilding the team with data involved from the startPrinciple 3: Processes to Support and AlignEstablishing a Knowledge BaselineEstablishing a Common VocabularyDeveloping a Rhythm Around Data Collection and SharingProject review meetingsSpreading data across the organizationCreating a Presence in the OfficeLearning from the PastSummaryQuestions to Ask Yourself

Ethical ConsiderationsEthics in Online ExperimentationDesign Experimentation Versus Social ExperimentationTwo “Power of Suggestion” ExperimentsToward Ethical A/B TestingKey ConceptsAsking Questions, Thinking EthicallyLast Words
KeywordsChapter 1Chapter 2Chapter 3Chapters 4, 5, and 6Chapter 7Chapter 8BooksOnline Articles, Papers, and BlogsCoursesToolsProfessional Groups, Meetups, and Societies

Content preview from Designing with Data

Chapter 6. The Analysis Phase (Getting Answers From Your Experiments)

LAUNCHING YOUR A/B TEST out “in the wild” is exciting—it’s the first time you’ll get to have a conversation with your users at scale about how your designs will serve them and meet their needs! This process of launching and learning from your test will be our focus in this chapter. One of the principles behind A/B testing is that it’s a good idea to always test first with a very small portion of your user base. The data you get back from your experiments will give you a sense of how well your design is performing with respect to your goals, before you invest the time to launch a design to all of your users. Did the design(s) lead to the behaviors you were expecting to see? Did you see that people acted in ways which map more to your business metrics? Were you able to successfully improve the metrics that you were targeting? For example, if your goal is that your users watch more movies they like, did you see an uptick in movies selected and watched all the way through? This would indicate that your recommendation algorithms and/or your presentation of selections are more effective than in your previous design.

Once you get back data from your initial tests, you then need to decide what you’ll do next. Do you take that experience and then roll it out to a larger part of your user base? Did the results suggest you are on the wrong track, that you should abandon the idea entirely? Or do you keep working on your hypothesis ...