Context Engineering with DSPy

Chapter 5. Formalizing Evaluation Metrics

A note for Early Release readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book. The GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In chapter 4 you learned strategies for collecting datasets. You saw how error analysis helps you identify common failure patterns in your program, and how to work with domain experts and language models to build a definitive list of typical inputs and expected outputs for your program. It was impressed upon you the importance of collecting useful examples of your task, from which we can learn the ‘rules’ of how your program should work. But a dataset alone isn’t enough, you need a formal way to measure whether your program is getting better or worse. That’s where evaluation metrics come in.

Evaluation is an expert topic, and the primary preoccupation of AI engineers who have a product in production. Ultimately, if you don’t have good eval metrics, you don’t know if your application is failing or succeeding, and can’t anticipate whether a change you want to make will help or harm your users. Context engineering starts and ends with evaluation: it’s how you notice there’s ...