Analytics the Right Way

Chapter 7Hypothesis Validation with New Evidence

In the last chapter, we focused on how to articulate, organize, and prioritize hypotheses. In this chapter and the next two chapters, we’ll dig into the techniques we can use to validate them and the various considerations around those techniques.

The first step is determining if new evidence is actually needed to validate the hypothesis. This is determined by answering three questions:

• What evidence do we already have?
• How strongly does that evidence alone already validate the hypothesis?
• How high are the stakes regarding the decision that the validation of the hypothesis is intended to inform?

By way of a silly (if deadly!) example, consider parachutes. We’ve never scientifically tested whether parachutes reduce the probability of death for skydivers,¹ but we’re pretty darn sure they work:

I believe wearing a parachute will reduce the probability I expire after my first skydive because parachutes decrease the speed at which I may crash into the earth, they’ve been used by almost everyone intentionally jumping out of an airplane ever since people started jumping out of airplanes, and because almost no one who hasn’t worn one has lived to tell their story the next day. If I’m right, I will wear a parachute when I jump out of an airplane.

Knowing whether parachutes are effective is a mission-critical piece of information for ...