97 Things About Ethics Everyone in Data Science Should Know

Chapter 18. Why Research Should Be Reproducible

Stuart Buck

Vice President of Research, Arnold Ventures

Today’s sciences—especially the social sciences—are in a bit of turmoil. Many of the most important experiments and findings are not reproducible. This “reproducibility crisis” has significant implications not just for the future of academic research and development but also for any business expecting increased returns from investing in innovation, experimentation, and data analysis. Business needs to learn from science’s mistakes.

As Vice President of Research at Arnold Ventures, I have close knowledge of this ongoing crisis, because I have funded a good deal of these “second look” efforts. Here’s an unhappy sample of what we funded and found:

• In 2015, the journal Science published the results of the largest replication project ever performed: the Reproducibility Project in Psychology, in which hundreds of researchers around the world attempted to replicate 100 psychology experiments from top journals. Only about 40% of the findings could be successfully replicated, while the rest were either inconclusive or definitively not replicated.

• In 2018, the Social Sciences Replication Project attempted to replicate 21 social science experiments that had been published in the journals Science and Nature between 2010 and 2015. Only 13 of the 21 experiments could be replicated successfully, ...