book

Designing with Data

by Rochelle King, Elizabeth F Churchill, Caitlin Tan

March 2017

Beginner to intermediate

370 pages

9h 55m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Praise for Designing with Data
Foreword
Preface
Design and Data: A Perfect SynergyOur Focus: A/B TestingSome Orienting PrinciplesWho Is This Book For?ScopeAbout UsA Word from RochelleA Word from ElizabethA Word from CaitlinHow This Book Is OrganizedHow to Read This BookIntroducing our “Running a Camp” MetaphorO’Reilly SafariHow to Contact UsAcknowledgmentsRochelleElizabethCaitlin
1. Introducing a Data Mindset
Data as a TrendThree Ways to Think About DataWhat Does This Mean for You as a Designer?Data Can Help to Align Design with BusinessOn Data QualityWith a Little Help from Your Friends...Data ProducersData ConsumersWhat If You Don’t Have Data Friends (Yet)?Themes You’ll See in This BookSummaryQuestions to Ask Yourself
2. The ABCs of Using Data
The Diversity of DataMany Dimensions of DataWhy are you collecting data?When is the data collected?How is the data collected?How much data to collect?Why Experiment?Learning About CausalityStatistically Significant, not AnecdotalInformed Opinions about what will happen in the WildBasics of ExperimentationLanguage and ConceptsRace to the Campsite!Experimentation in the Internet AgeA/B Testing: Online ExperimentsSampling Your Users OnlineCohorts and segmentsDemographic informationNew users versus existing usersMetrics: The Dependent Variable of A/B TestingDetecting a Difference in Your GroupsHow big is the difference you want to measure?A big enough sample to power your testSignificance levelYour Hypothesis and Why It MattersDefining a Hypothesis or HypothesesKnow What You Want to LearnRunning Creative A/B TestsData Triangulation: Strength in Mixed MethodsThe Landscape of Design ActivitiesExploring and evaluating IdeasThinking Global and Thinking LocalSummaryQuestions to Ask Yourself
3. A Framework for Experimentation
Introducing Our FrameworkWorking with Data Should Feel Familiar...Three Phases: Definition, Execution, and AnalysisThe Definition PhaseThe Execution PhaseThe Analysis PhaseExamples: Data and Design in ActionSummaryQuestions to Ask Yourself
4. The Definition Phase (How to Frame Your Experiments)
Getting Started: Defining Your GoalDefining Your Metric of InterestMetric sensitivityTracking multiple metricsGetting the full pictureYour metrics may change over timeCompeting metricsRefining Your Goals with DataIdentifying the Problem You Are SolvingRemember Where You AreBuilding Hypotheses for the Problem at HandExample: A Summer Camp HypothesisExample: Netflix—transitioning from DVD rentals to StreamingThe Importance of Going BroadMultiple Ways to Influence a MetricFocus on New and Existing UsersRevisit the Scope of Your ProblemExample: Netflix on the PlayStation 3Involve Your Team and Your Data FriendsWhich Hypotheses to Choose?Consider Potential ImpactUsing What You Already KnowUsing Other Methods to Evaluate Your HypothesesConsider the Reality of Your TestHow much measurable impact do you believe your hypothesis can make?Can you draw all the conclusions you want to draw from your test?Balancing learning and speedKeep Your Old Hypotheses in Your Back PocketSummaryQuestions to Ask Yourself
5. The Execution Phase (How to Put Your Experiments into Action)
Designing to LearnEngaging Your Users in a ConversationHaving Quality ConversationsDesigning to extremes to learn about your usersRevisiting the minimum detectable effectDesigning the Best Representation of Your HypothesisUnderstanding Your VariablesNot all variables are visibleYour Design Can Influence Your DataExample: Netflix WiiRevisiting the Space of Design ActivitiesAvoiding Local MaximaDifferent problems for summer campDirectional testing: “Painted door” testsPicking the right level of granularity for your experimentExample: Netflix on Playstation 3Example: Spotify NavigationExperiment 1: Defining the hypothesis to get early directional feedbackExperiment 1: Designing the hypothesesInterlude: Quick explorations using prototypes and usability testingExperiment 2: Refining the “tabbed” navigation“Designing” your testsOther Considerations When Designing to LearnPolishing your design too much, too earlyEdge cases and “worst-case” scenariosTaking advantage of other opportunities to learn about your designIdentifying the Right Level of Testing for Different Stages of ExperimentationRunning parallel experimentsThinking about “Experiment 0”SummaryQuestions to Ask Yourself
6. The Analysis Phase (Getting Answers From Your Experiments)
Vetting Your Designs Ahead of LaunchLab Studies: Interviews and Usability TestingSurveysWorking with Your Peers in DataLaunching Your DesignBalancing Trade Offs to Power Your TestWeighing sample size and significance levelGetting the sample that you need (rollout % versus test time)Who are you including in your sample?Practical Implementation DetailsIs your experience “normal” right now?Sanity check: Questions to ask yourselfEvaluating Your ResultsRevisiting Statistical SignificanceWhat Does the Data Say?Expected (“Positive”) ResultsUnexpected and Undesirable (“Negative”) ResultsWhen the World is FlatErrorsReplicationUsing secondary metricsUsing multiple test cellsRolling out to more usersRevisiting “thick” dataGetting Trustworthy DataNovelty effectSeasonality biasRolling Out Your Experience, or NotWhat’s Next for Your Designs?Were you exploring or evaluating?Was your problem global or local?Knowing when to stopRamp UpHoldback GroupsTaking Communication into AccountCase Study: Netflix on PlayStation 3Many Treatments of the Four HypothesesEvolving the Design Through Iterative TestsWhat If You Still Believe?SummaryQuestions to Ask Yourself
7. Creating the Right Environment for Data-Aware Design
Principle 1: Shared Company Culture and ValuesDepth: Communicating Across LevelsBreadth: Beyond Design and ProductThe Importance of a Learning CultureThe rewards of taking risks: Redefining “failure”The value of developing your customer instinctPrinciple 2: Hiring and Growing the Right PeopleEstablishing a Data-Aware Environment Through Your PeersHiring for SuccessBuilding the team with data involved from the startPrinciple 3: Processes to Support and AlignEstablishing a Knowledge BaselineEstablishing a Common VocabularyDeveloping a Rhythm Around Data Collection and SharingProject review meetingsSpreading data across the organizationCreating a Presence in the OfficeLearning from the PastSummaryQuestions to Ask Yourself

8. Conclusion
Ethical ConsiderationsEthics in Online ExperimentationDesign Experimentation Versus Social ExperimentationTwo “Power of Suggestion” ExperimentsToward Ethical A/B TestingKey ConceptsAsking Questions, Thinking EthicallyLast Words
A. Resources
KeywordsChapter 1Chapter 2Chapter 3Chapters 4, 5, and 6Chapter 7Chapter 8BooksOnline Articles, Papers, and BlogsCoursesToolsProfessional Groups, Meetups, and Societies
Index

Content preview from Designing with Data

Appendix A. Resources

THIS APPENDIX LISTS SOME additional resources that you may find useful and/or interesting as you dive deeper into experimentation, or into user experience research around your designs.

Keywords

We all know one of the best ways to learn about an area is to start with some keyword searches online. Here are some keywords to get you started, listed chapter by chapter:

Chapter 1

Keywords: data analytics, data science, data visualization, design thinking, experimental design, introduction to statistics, mixed methods, reasoning biases, user experience research

Chapter 2

Keywords: acquiescence bias, attitudinal data, correlation, causality, cohorts, data triangulation, demographics, dependent variable, detectable difference, independent variable, local versus global effects, metric, Philosophy of Science, population, qualitative data, quantitative data, research methods, samples, segments, social desirability bias, statistical significance, test users

Chapter 3

Keywords: design opportunity, experimentation framework, hypothesis statement

Chapters 4, 5, and 6

Keywords: A/B testing, big data, daily active users, errors, experimentation, lab studies, metric of interest, metric sensitivity, metrics, minimal detectable effect (MDE), multiple hypotheses, negative results, novelty effect, pilot studies, positive results, proxy metrics, sample, seasonal bias, secondary metrics, surveys, test cells, thick data, thin data, usability testing

Chapter 7

Keywords: collaborative teams, common ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781449334925Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills