Machine Learning in Production: Developing and Optimizing Data Science Workflows and Applications

3. Quantifying Error

To kill an error is as good a service as, and sometimes evenbetter than, the establishing of a new truth or fact.

—Charles Darwin

3.1 Introduction

Most measurements have some error associated with them. We often think of the numbers we report as exact values (e.g., “there were 9,126 views of this article”). Anyone who has implemented multiple tracking systems that are supposed to measure the same quantity knows there is rarely perfect agreement between measurements. The chances are that neither system measures the ground truth—there are always failure modes, and it’s hard to know how often failures happen.

Aside from errors in data collections, some measured quantities are uncertain. Instead of running an experiment with ...