book

Designing with Data

by Rochelle King, Elizabeth F Churchill, Caitlin Tan

March 2017

Beginner to intermediate

370 pages

9h 55m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Design and Data: A Perfect SynergyOur Focus: A/B TestingSome Orienting PrinciplesWho Is This Book For?ScopeAbout UsA Word from RochelleA Word from ElizabethA Word from CaitlinHow This Book Is OrganizedHow to Read This BookIntroducing our “Running a Camp” MetaphorO’Reilly SafariHow to Contact UsAcknowledgmentsRochelleElizabethCaitlin
Data as a TrendThree Ways to Think About DataWhat Does This Mean for You as a Designer?Data Can Help to Align Design with BusinessOn Data QualityWith a Little Help from Your Friends...Data ProducersData ConsumersWhat If You Don’t Have Data Friends (Yet)?Themes You’ll See in This BookSummaryQuestions to Ask Yourself
The Diversity of DataMany Dimensions of DataWhy are you collecting data?When is the data collected?How is the data collected?How much data to collect?Why Experiment?Learning About CausalityStatistically Significant, not AnecdotalInformed Opinions about what will happen in the WildBasics of ExperimentationLanguage and ConceptsRace to the Campsite!Experimentation in the Internet AgeA/B Testing: Online ExperimentsSampling Your Users OnlineCohorts and segmentsDemographic informationNew users versus existing usersMetrics: The Dependent Variable of A/B TestingDetecting a Difference in Your GroupsHow big is the difference you want to measure?A big enough sample to power your testSignificance levelYour Hypothesis and Why It MattersDefining a Hypothesis or HypothesesKnow What You Want to LearnRunning Creative A/B TestsData Triangulation: Strength in Mixed MethodsThe Landscape of Design ActivitiesExploring and evaluating IdeasThinking Global and Thinking LocalSummaryQuestions to Ask Yourself
Introducing Our FrameworkWorking with Data Should Feel Familiar...Three Phases: Definition, Execution, and AnalysisThe Definition PhaseThe Execution PhaseThe Analysis PhaseExamples: Data and Design in ActionSummaryQuestions to Ask Yourself
Getting Started: Defining Your GoalDefining Your Metric of InterestMetric sensitivityTracking multiple metricsGetting the full pictureYour metrics may change over timeCompeting metricsRefining Your Goals with DataIdentifying the Problem You Are SolvingRemember Where You AreBuilding Hypotheses for the Problem at HandExample: A Summer Camp HypothesisExample: Netflix—transitioning from DVD rentals to StreamingThe Importance of Going BroadMultiple Ways to Influence a MetricFocus on New and Existing UsersRevisit the Scope of Your ProblemExample: Netflix on the PlayStation 3Involve Your Team and Your Data FriendsWhich Hypotheses to Choose?Consider Potential ImpactUsing What You Already KnowUsing Other Methods to Evaluate Your HypothesesConsider the Reality of Your TestHow much measurable impact do you believe your hypothesis can make?Can you draw all the conclusions you want to draw from your test?Balancing learning and speedKeep Your Old Hypotheses in Your Back PocketSummaryQuestions to Ask Yourself
Designing to LearnEngaging Your Users in a ConversationHaving Quality ConversationsDesigning to extremes to learn about your usersRevisiting the minimum detectable effectDesigning the Best Representation of Your HypothesisUnderstanding Your VariablesNot all variables are visibleYour Design Can Influence Your DataExample: Netflix WiiRevisiting the Space of Design ActivitiesAvoiding Local MaximaDifferent problems for summer campDirectional testing: “Painted door” testsPicking the right level of granularity for your experimentExample: Netflix on Playstation 3Example: Spotify NavigationExperiment 1: Defining the hypothesis to get early directional feedbackExperiment 1: Designing the hypothesesInterlude: Quick explorations using prototypes and usability testingExperiment 2: Refining the “tabbed” navigation“Designing” your testsOther Considerations When Designing to LearnPolishing your design too much, too earlyEdge cases and “worst-case” scenariosTaking advantage of other opportunities to learn about your designIdentifying the Right Level of Testing for Different Stages of ExperimentationRunning parallel experimentsThinking about “Experiment 0”SummaryQuestions to Ask Yourself
Vetting Your Designs Ahead of LaunchLab Studies: Interviews and Usability TestingSurveysWorking with Your Peers in DataLaunching Your DesignBalancing Trade Offs to Power Your TestWeighing sample size and significance levelGetting the sample that you need (rollout % versus test time)Who are you including in your sample?Practical Implementation DetailsIs your experience “normal” right now?Sanity check: Questions to ask yourselfEvaluating Your ResultsRevisiting Statistical SignificanceWhat Does the Data Say?Expected (“Positive”) ResultsUnexpected and Undesirable (“Negative”) ResultsWhen the World is FlatErrorsReplicationUsing secondary metricsUsing multiple test cellsRolling out to more usersRevisiting “thick” dataGetting Trustworthy DataNovelty effectSeasonality biasRolling Out Your Experience, or NotWhat’s Next for Your Designs?Were you exploring or evaluating?Was your problem global or local?Knowing when to stopRamp UpHoldback GroupsTaking Communication into AccountCase Study: Netflix on PlayStation 3Many Treatments of the Four HypothesesEvolving the Design Through Iterative TestsWhat If You Still Believe?SummaryQuestions to Ask Yourself
Principle 1: Shared Company Culture and ValuesDepth: Communicating Across LevelsBreadth: Beyond Design and ProductThe Importance of a Learning CultureThe rewards of taking risks: Redefining “failure”The value of developing your customer instinctPrinciple 2: Hiring and Growing the Right PeopleEstablishing a Data-Aware Environment Through Your PeersHiring for SuccessBuilding the team with data involved from the startPrinciple 3: Processes to Support and AlignEstablishing a Knowledge BaselineEstablishing a Common VocabularyDeveloping a Rhythm Around Data Collection and SharingProject review meetingsSpreading data across the organizationCreating a Presence in the OfficeLearning from the PastSummaryQuestions to Ask Yourself

Ethical ConsiderationsEthics in Online ExperimentationDesign Experimentation Versus Social ExperimentationTwo “Power of Suggestion” ExperimentsToward Ethical A/B TestingKey ConceptsAsking Questions, Thinking EthicallyLast Words
KeywordsChapter 1Chapter 2Chapter 3Chapters 4, 5, and 6Chapter 7Chapter 8BooksOnline Articles, Papers, and BlogsCoursesToolsProfessional Groups, Meetups, and Societies

Content preview from Designing with Data

Foreword

This book offers a simple but exciting promise for anyone involved with the design of internet products:

If you compare what you think should happen (in theory) to what actually happens (in reality), you can better understand how your work changes user behavior.

This book is your training camp to getting started using data—design’s often overlooked but essential companion—to improve your work. You can augment this approach with your existing design practices to amplify the impact of each design: from identifying the problem you are solving, to designing for experimentation, to learning from the data.

What’s unique about this training camp are the field guides at hand: not only your authors sharing experiences from the world’s leading data-driven organizations, but also the contributions from many practitioners offering you the advantage of their standard practices, case studies, and hard-fought lessons from designing with data every single day.

COLIN MCFARLAND,

HEAD OF EXPERIMENTATION AT SKYSCANNER