Data Science: The Hard Parts

Chapter 2. Metrics Design

Let me propose that great data scientists are also great at metrics design. What is metrics design? A short answer is that it is the art and science of finding metrics with good properties. I will discuss some of these desirable properties shortly, but first let me make a case for why data scientists ought to be great at it.

A simple answer is: because if not us, who else? Ideally everyone at the organization should excel at metrics design. But data practitioners are the best fit for that task. Data scientists work with metrics all the time: they calculate, report, analyze, and, hopefully, attempt to optimize them. Take A/B testing: the starting point of every good test is having the right output metric. A similar rationale applies for machine learning (ML): getting the correct outcome metric to predict is of utmost importance.

Desirable Properties That Metrics Should Have

Why do companies need metrics? As argued in Chapter 1, good metrics are there to drive actions. With this success criterion in mind, let’s reverse engineer the problem and identify necessary conditions for success.

Measurable

Metrics are measurable by definition. Unfortunately, many metrics are imperfect, and learning to identify their pitfalls will take you a long way. So-called proxy metrics or proxies that are usually correlated to the desired outcome abound, and you need to understand the pros and cons of working with them.¹

A simple example is intentionality. Suppose you want ...