Designing for Behavior Change, 2nd Edition

Chapter 14. Determining Impact When You Can’t Run an A/B Test

Recently I had the good fortune to attend an event featuring many of the most prominent and prolific behavioral scientists in the world. They had banded together to address one of the frontiers of behavioral science: long-term behavior change.

The researchers were testing whether they could move the needle on gym attendance. They ran nearly 20 simultaneous studies, with each researcher pursuing their “best guess” of what would work. This event was the first time that everyone in the group would hear the results.

What happened? Not a single study was effective at long-term behavior change. The best researchers in the world had each taken a shot and had fallen short of their target.

Something that is often hidden, at least to those outside of the research community, is that in our space the successes are few and the failures are many. That’s normal. That’s expected. It’s because the most interesting applications of behavioral science are often focused on difficult and seemingly intractable problems, like exercise. Even the best researchers in the world try again and again, before they find a breakthrough that generates headlines and a best-selling book.¹

The lesson here isn’t merely to “fail fast” (a motto that is deeply engrained in product development, at least in Silicon Valley and similar tech hubs) or that embracing ...