book

Head First Data Analysis

by Michael Milton

July 2009

Intermediate to advanced

483 pages

10h 38m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Who is this book for?Who should probably back away from this book?We know what you’re thinkingWe know what your brain is thinkingMetacognition: thinking about thinkingHere’s what WE didHere’s what YOU can do to bend your brain into submissionRead MeThe technical review teamAcknowledgmentsSafari® Books Online
Acme Cosmetics needs your helpThe CEO wants data analysis to help increase salesData analysis is careful thinking about evidenceDefine the problemYour client will help you define your problemAcme’s CEO has some feedback for youBreak the problem and data into smaller piecesDivide the problem into smaller problemsDivide the data into smaller chunksNow take another look at what you knowEvaluate the piecesAnalysis begins when you insert yourselfMake a recommendationYour report is readyThe CEO likes your workAn article just came across the wireYou let the CEO’s beliefs take you down the wrong pathYour assumptions and beliefs about the world are your mental modelYour statistical model depends on your mental modelMental models should always include what you don’t knowThe CEO tells you what he doesn’t knowAcme just sent you a huge list of raw dataTime to drill further into the dataGeneral American Wholesalers confirms your impressionHere’s what you didYour analysis led your client to a brilliant decision
It’s a coffee recession!The Starbuzz board meeting is in three monthsThe Starbuzz SurveyAlways use the method of comparisonComparisons are key for observational dataCould value perception be causing the revenue decline?A typical customer’s thinkingObservational studies are full of confoundersHow location might be confounding your resultsManage confounders by breaking the data into chunksIt’s worse than we thought!You need an experiment to say which strategy will work bestThe Starbuzz CEO is in a big hurryStarbuzz drops its pricesOne month later...Control groups give you a baselineNot getting fired 101Let’s experiment for real!One month later...Confounders also plague experimentsAvoid confounders by selecting groups carefullyRandomization selects similar groupsYour experiment is ready to goThe results are inStarbuzz has an empirically tested sales strategy
You’re now in the bath toy gameConstraints limit the variables you controlDecision variables are things you can controlYou have an optimization problemFind your objective with the objective functionYour objective functionShow product mixes with your other constraintsPlot multiple constraints on the same chartYour good options are all in the feasible regionYour new constraint changed the feasible regionYour spreadsheet does optimizationSolver crunched your optimization problem in a snapProfits fell through the floorYour model only describes what you put into itCalibrate your assumptions to your analytical objectivesWatch out for negatively linked variablesYour new plan is working like a charmYour assumptions are based on an ever-changing reality

New Army needs to optimize their websiteThe results are in, but the information designer is outThe last information designer submitted these three infographicsWhat data is behind the visualizations?Show the data!Here’s some unsolicited advice from the last designerToo much data is never your problemMaking the data pretty isn’t your problem eitherData visualization is all about making the right comparisonsYour visualization is already more useful than the rejected onesUse scatterplots to explore causesThe best visualizations are highly multivariateShow more variables by looking at charts togetherThe visualization is great, but the web guru’s not satisfied yetGood visual designs help you think about causesThe experiment designers weigh inThe experiment designers have some hypotheses of their ownThe client is pleased with your workOrders are coming in from everywhere!
Gimme some skin...When do we start making new phone skins?PodPhone doesn’t want you to predict their next moveHere’s everything we knowElectroSkinny’s analysis does fit the dataElectroSkinny obtained this confidential strategy memoVariables can be negatively or positively linkedCauses in the real world are networked, not linearHypothesize PodPhone’s optionsYou have what you need to run a hypothesis testFalsification is the heart of hypothesis testingDiagnosticity helps you find the hypothesis with the least disconfirmationYou can’t rule out all the hypotheses, but you can say which is strongestYou just got a picture message...It’s a launch!
The doctor has disturbing newsLet’s take the accuracy analysis one claim at a timeHow common is lizard flu really?You’ve been counting false positivesThe opposite of a false positive is a true negativeAll these terms describe conditional probabilitiesYou need to count1 percent of people have lizard fluWatch out for the base rate fallacyYour chances of having lizard flu are still pretty lowDo complex probabilistic thinking with simple whole numbersBayes’ rule manages your base rates when you get new dataYou can use Bayes’ rule over and overYour second test result is negativeThe new test has different accuracy statisticsNew information can change your base rateWhat a relief!
Backwater Investments needs your helpTheir analysts are at each other’s throatsSubjective probabilities describe expert beliefsSubjective probabilities might show no real disagreement after allThe analysts responded with their subjective probabilitiesThe CEO doesn’t see what you’re up toThe CEO loves your workThe standard deviation measures how far points are from the averageYou were totally blindsided by this newsBayes’ rule is great for revising subjective probabilitiesThe CEO knows exactly what to do with this new informationRussian stock owners rejoice!
LitterGitters submitted their report to the city councilThe LitterGitters have really cleaned up this townThe LitterGitters have been measuring their campaign’s effectivenessThe mandate is to reduce the tonnage of litterTonnage is unfeasible to measureGive people a hard question, and they’ll answer an easier one insteadLittering in Dataville is a complex systemYou can’t build and implement a unified litter-measuring modelHeuristics are a middle ground between going with your gut and optimizationUse a fast and frugal treeIs there a simpler way to assess LitterGitters’ success?Stereotypes are heuristicsYour analysis is ready to presentLooks like your analysis impressed the city council members
Your annual review is coming upGoing for more cash could play out in a bunch of different waysHere’s some data on raisesHistograms show frequencies of groups of numbersGaps between bars in a histogram mean gaps among the data pointsInstall and run RLoad data into RR creates beautiful histogramsMake histograms from subsets of your dataNegotiation paysWhat will negotiation mean for you?
What are you going to do with all this money?An analysis that tells people what to ask for could be hugeBehold... the Raise Reckoner!Inside the algorithm will be a method to predict raisesScatterplots compare two variablesA line could tell your clients where to aimPredict values in each strip with the graph of averagesThe regression line predicts what raises people will receiveThe line is useful if your data shows a linear correlationYou need an equation to make your predictions precisea represents the y-axis interceptb represents the slopeTell R to create a regression objectThe regression equation goes hand in hand with your scatterplotThe regression equation is the Raise Reckoner algorithmYour raise predictor didn’t work out as planned...
Your clients are pretty ticked offWhat did your raise prediction algorithm do?The segments of customersThe guy who asked for 25% went outside the modelHow to handle the client who wants a prediction outside the data rangeThe guy who got fired because of extrapolation has cooled offYou’ve only solved part of the problemWhat does the data for the screwy outcomes look like?Chance errors are deviations from what your model predictsError is good for you and your clientSpecify error quantitativelyQuantify your residual distribution with Root Mean Squared errorYour model in R already knows the R.M.S. errorR’s summary of your linear model shows your R.M.S. errorSegmentation is all about managing errorGood regressions balance explanation and predictionYour segmented models manage error better than the original modelYour clients are returning in droves
The Dataville Dispatch wants to analyze salesHere’s the data they keep to track their operationsYou need to know how the data tables relate to each otherA database is a collection of data with well-specified relations to each otherTrace a path through the relations to make the comparison you needCreate a spreadsheet that goes across that pathYour summary ties article count and sales togetherLooks like your scatterplot is going over really wellCopying and pasting all that data was a painRelational databases manage relations for youDataville Dispatch built an RDBMS with your relationship diagramDataville Dispatch extracted your data using the SQL languageComparison possibilities are endless if your data is in a RDBMSYou’re on the cover
Just got a client list from a defunct competitorThe dirty secret of data analysisHead First Head Hunters wants the list for their sales teamCleaning messy data is all about preparationOnce you’re organized, you can fix the data itselfUse the # sign as a delimiterExcel split your data into columns using the delimiterUse SUBSTITUTE to replace the carat characterYou cleaned up all the first namesThe last name pattern is too complex for SUBSTITUTEHandle complex patterns with nested text formulasR can use regular expressions to crunch complex data patternsThe sub command fixed your last namesNow you can ship the data to your clientMaybe you’re not quite done yet...Sort your data to show duplicate values togetherThe data is probably from a relational databaseRemove duplicate namesYou created nice, clean, unique recordsHead First Head Hunters is recruiting like gangbusters!Leaving town...It’s been great having you here in Dataville!
#1: Everything else in statistics#2: Excel skills#3: Edward Tufte and his principles of visualization#4: PivotTables#5: The R community#6: Nonlinear and multiple regression#7: Null-alternative hypothesis testing#8: Randomness#9: Google Docs#10: Your expertise
Get started with R
Install the data analysis tools in Excel

Content preview from Head First Data Analysis

Chapter 8. Heuristics: Analyze like a human

The real world has more variables than you can handle.

There is always going to be data that you can’t have. And even when you do have data on most of the things you want to understand, optimizing methods are often elusive and time consuming. Fortunately, most of the actual thinking you do in life is not “rational maximizing”—it’s processing incomplete and uncertain information with rules of thumb so that you can make decisions quickly. What is really cool is that these rules can actually work and are important (and necessary) tools for data analysts.

LitterGitters submitted their report to the city council

The LitterGitters are a nonprofit group funded by the Dataville City Council to run public service announcements to encourage people to stop littering.

They just presented the results of their most recent work to the city council, and the reaction is not what they’d hoped for.

That last comment is the one we’re really worried about. It’s starting to look as if LitterGitters will be in big trouble very soon if you can’t persuade the city council that LitterGitters’ public outreach programs have been a success relative to the city council’s intentions for it.