book

Designing with Data

by Rochelle King, Elizabeth F Churchill, Caitlin Tan

March 2017

Beginner to intermediate

370 pages

9h 55m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Design and Data: A Perfect SynergyOur Focus: A/B TestingSome Orienting PrinciplesWho Is This Book For?ScopeAbout UsA Word from RochelleA Word from ElizabethA Word from CaitlinHow This Book Is OrganizedHow to Read This BookIntroducing our “Running a Camp” MetaphorO’Reilly SafariHow to Contact UsAcknowledgmentsRochelleElizabethCaitlin
Data as a TrendThree Ways to Think About DataWhat Does This Mean for You as a Designer?Data Can Help to Align Design with BusinessOn Data QualityWith a Little Help from Your Friends...Data ProducersData ConsumersWhat If You Don’t Have Data Friends (Yet)?Themes You’ll See in This BookSummaryQuestions to Ask Yourself
The Diversity of DataMany Dimensions of DataWhy are you collecting data?When is the data collected?How is the data collected?How much data to collect?Why Experiment?Learning About CausalityStatistically Significant, not AnecdotalInformed Opinions about what will happen in the WildBasics of ExperimentationLanguage and ConceptsRace to the Campsite!Experimentation in the Internet AgeA/B Testing: Online ExperimentsSampling Your Users OnlineCohorts and segmentsDemographic informationNew users versus existing usersMetrics: The Dependent Variable of A/B TestingDetecting a Difference in Your GroupsHow big is the difference you want to measure?A big enough sample to power your testSignificance levelYour Hypothesis and Why It MattersDefining a Hypothesis or HypothesesKnow What You Want to LearnRunning Creative A/B TestsData Triangulation: Strength in Mixed MethodsThe Landscape of Design ActivitiesExploring and evaluating IdeasThinking Global and Thinking LocalSummaryQuestions to Ask Yourself
Introducing Our FrameworkWorking with Data Should Feel Familiar...Three Phases: Definition, Execution, and AnalysisThe Definition PhaseThe Execution PhaseThe Analysis PhaseExamples: Data and Design in ActionSummaryQuestions to Ask Yourself
Getting Started: Defining Your GoalDefining Your Metric of InterestMetric sensitivityTracking multiple metricsGetting the full pictureYour metrics may change over timeCompeting metricsRefining Your Goals with DataIdentifying the Problem You Are SolvingRemember Where You AreBuilding Hypotheses for the Problem at HandExample: A Summer Camp HypothesisExample: Netflix—transitioning from DVD rentals to StreamingThe Importance of Going BroadMultiple Ways to Influence a MetricFocus on New and Existing UsersRevisit the Scope of Your ProblemExample: Netflix on the PlayStation 3Involve Your Team and Your Data FriendsWhich Hypotheses to Choose?Consider Potential ImpactUsing What You Already KnowUsing Other Methods to Evaluate Your HypothesesConsider the Reality of Your TestHow much measurable impact do you believe your hypothesis can make?Can you draw all the conclusions you want to draw from your test?Balancing learning and speedKeep Your Old Hypotheses in Your Back PocketSummaryQuestions to Ask Yourself
Designing to LearnEngaging Your Users in a ConversationHaving Quality ConversationsDesigning to extremes to learn about your usersRevisiting the minimum detectable effectDesigning the Best Representation of Your HypothesisUnderstanding Your VariablesNot all variables are visibleYour Design Can Influence Your DataExample: Netflix WiiRevisiting the Space of Design ActivitiesAvoiding Local MaximaDifferent problems for summer campDirectional testing: “Painted door” testsPicking the right level of granularity for your experimentExample: Netflix on Playstation 3Example: Spotify NavigationExperiment 1: Defining the hypothesis to get early directional feedbackExperiment 1: Designing the hypothesesInterlude: Quick explorations using prototypes and usability testingExperiment 2: Refining the “tabbed” navigation“Designing” your testsOther Considerations When Designing to LearnPolishing your design too much, too earlyEdge cases and “worst-case” scenariosTaking advantage of other opportunities to learn about your designIdentifying the Right Level of Testing for Different Stages of ExperimentationRunning parallel experimentsThinking about “Experiment 0”SummaryQuestions to Ask Yourself
Vetting Your Designs Ahead of LaunchLab Studies: Interviews and Usability TestingSurveysWorking with Your Peers in DataLaunching Your DesignBalancing Trade Offs to Power Your TestWeighing sample size and significance levelGetting the sample that you need (rollout % versus test time)Who are you including in your sample?Practical Implementation DetailsIs your experience “normal” right now?Sanity check: Questions to ask yourselfEvaluating Your ResultsRevisiting Statistical SignificanceWhat Does the Data Say?Expected (“Positive”) ResultsUnexpected and Undesirable (“Negative”) ResultsWhen the World is FlatErrorsReplicationUsing secondary metricsUsing multiple test cellsRolling out to more usersRevisiting “thick” dataGetting Trustworthy DataNovelty effectSeasonality biasRolling Out Your Experience, or NotWhat’s Next for Your Designs?Were you exploring or evaluating?Was your problem global or local?Knowing when to stopRamp UpHoldback GroupsTaking Communication into AccountCase Study: Netflix on PlayStation 3Many Treatments of the Four HypothesesEvolving the Design Through Iterative TestsWhat If You Still Believe?SummaryQuestions to Ask Yourself
Principle 1: Shared Company Culture and ValuesDepth: Communicating Across LevelsBreadth: Beyond Design and ProductThe Importance of a Learning CultureThe rewards of taking risks: Redefining “failure”The value of developing your customer instinctPrinciple 2: Hiring and Growing the Right PeopleEstablishing a Data-Aware Environment Through Your PeersHiring for SuccessBuilding the team with data involved from the startPrinciple 3: Processes to Support and AlignEstablishing a Knowledge BaselineEstablishing a Common VocabularyDeveloping a Rhythm Around Data Collection and SharingProject review meetingsSpreading data across the organizationCreating a Presence in the OfficeLearning from the PastSummaryQuestions to Ask Yourself

Ethical ConsiderationsEthics in Online ExperimentationDesign Experimentation Versus Social ExperimentationTwo “Power of Suggestion” ExperimentsToward Ethical A/B TestingKey ConceptsAsking Questions, Thinking EthicallyLast Words
KeywordsChapter 1Chapter 2Chapter 3Chapters 4, 5, and 6Chapter 7Chapter 8BooksOnline Articles, Papers, and BlogsCoursesToolsProfessional Groups, Meetups, and Societies

Content preview from Designing with Data

Appendix A. Resources

THIS APPENDIX LISTS SOME additional resources that you may find useful and/or interesting as you dive deeper into experimentation, or into user experience research around your designs.

Keywords

We all know one of the best ways to learn about an area is to start with some keyword searches online. Here are some keywords to get you started, listed chapter by chapter:

Chapter 1

Keywords: data analytics, data science, data visualization, design thinking, experimental design, introduction to statistics, mixed methods, reasoning biases, user experience research

Chapter 2

Keywords: acquiescence bias, attitudinal data, correlation, causality, cohorts, data triangulation, demographics, dependent variable, detectable difference, independent variable, local versus global effects, metric, Philosophy of Science, population, qualitative data, quantitative data, research methods, samples, segments, social desirability bias, statistical significance, test users

Chapter 3

Keywords: design opportunity, experimentation framework, hypothesis statement

Chapters 4, 5, and 6

Keywords: A/B testing, big data, daily active users, errors, experimentation, lab studies, metric of interest, metric sensitivity, metrics, minimal detectable effect (MDE), multiple hypotheses, negative results, novelty effect, pilot studies, positive results, proxy metrics, sample, seasonal bias, secondary metrics, surveys, test cells, thick data, thin data, usability testing

Chapter 7

Keywords: collaborative teams, common ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Publisher Resources

ISBN: 9781449334925Errata Page

Designing with Data

by Rochelle King, Elizabeth F Churchill, Caitlin Tan

Appendix A. Resources

Keywords

Chapter 1

Chapter 2

Chapter 3

Chapters 4, 5, and 6

Chapter 7

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

You might also like

Storytelling with Data

Generative AI Design Patterns

Communicating with Data

Data Mesh

Publisher Resources

Appendix A. Resources

Keywords

Chapter 1

Chapter 2

Chapter 3

Chapters 4, 5, and 6

Chapter 7

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,and much more.

You might also like

Storytelling with Data

Generative AI Design Patterns

Communicating with Data

Data Mesh

Publisher Resources

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.