Statistics Done Wrong

Chapter 6. Double-Dipping in the Data

Earlier, we discussed truth inflation, a symptom of the overuse of significance testing. In the quest for significance, researchers select only the luckiest and most exaggerated results since those are the only ones that pass the significance filter. But that’s not the only way research gets biased toward exaggerated results.

Statistical analyses are often exploratory. In exploratory data analysis, you don’t choose a hypothesis to test in advance. You collect data and poke it to see what interesting details might pop out, ideally leading to new hypotheses and new experiments. This process involves making numerous plots, trying a few statistical analyses, and following any promising leads.

But aimlessly exploring ...