Lean Analytics

Chapter 4. Data-Driven Versus Data-Informed

Data is a powerful thing. It can be addictive, making you overanalyze everything. But much of what we actually do is unconscious, based on past experience and pragmatism. And with good reason: relying on wisdom and experience, rather than rigid analysis, helps us get through our day. After all, you don’t run A/B testing before deciding what pants to put on in the morning; if you did, you’d never get out the door.

One of the criticisms of Lean Startup is that it’s too data-driven. Rather than be a slave to the data, these critics say, we should use it as a tool. We should be data-informed, not data-driven. Mostly, they’re just being lazy, and looking for reasons not to do the hard work. But sometimes, they have a point: using data to optimize one part of your business, without stepping back and looking at the big picture, can be dangerous—even fatal.

Consider travel agency Orbitz and its discovery that Mac users were willing to reserve a more expensive hotel room. CTO Roger Liew told the Wall Street Journal, “We had the intuition [that Mac users are 40% more likely to book a four- or five-star hotel than PC users and to stay in more expensive rooms], and we were able to confirm it based on the data.”^[14]

On the one hand, an algorithm that ignores seemingly unrelated customer data (in this case, whether visitors were using a Mac) wouldn’t have found this opportunity to increase revenues. On the other hand, an algorithm that blindly optimizes ...