Beyond Code

Chapter 11

The Goodhart Trap

The dysfunction experiments used a scoring system with seven metrics. Each metric measured a specific aspect of the system’s quality: whether the services implemented the required endpoints, whether the data models were correct, whether the API contracts were consistent, whether the error handling followed the specification. The scoring system was well-designed, internally consistent, and more rigorous than most evaluation frameworks used in production software organizations. A perfect score was twenty-eight, and the single agent achieved it. The pipeline scored zero. These numbers are useful. They are also, in a specific and important sense, misleading.

The scoring system measured whether the implementation conformed ...