I’m often reminded that I’m human. Sometimes, this reminder comes to me in the form of small mishaps—coffee spilled on a white carpet, an insensitive remark made to a friend, spinach stuck between my teeth—things I can correct with little embarrassment to myself. All too often, however, this reminder isn’t nearly so gentle. Sometimes it hits me like a brick to the forehead. Like when I’m standing in front of a room full of important people, declaring the brilliance of my statistical conclusions to anyone who will listen.

My first data analysis humiliation occurred when I was only a few weeks into my career as a research statistician. I was working for a biomedical research company, call it MajorMedicCorp. This company developed new devices for monitoring a patient’s blood chemistry. This company wanted to measure certain blood toxins noninvasively, meaning without drawing any blood. We knew we could measure how much life-giving oxygen was in the blood, simply by shining light through a patient’s finger and measuring what came out. My task was to use similar light measurements to predict the amount of this toxic substance.

I was fresh out of school with a Ph.D. and lots of fancy statistical techniques at my fingertips, and this was a linear regression problem. Linear regression is undergraduate stuff. By all accounts, this task should’ve been a breeze for me. Yet somehow, I still managed to turn this ...

Get Beyond Basic Statistics: Tips, Tricks, and Techniques Every Data Analyst Should Know now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.