Large Language Models: The Hard Parts

Chapter 2. The Evals Gap

It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with experiment, it’s wrong.

Richard Feynman

The advent of LLMs and the subsequent rapid adoption of LLMBAs mark a pivotal shift in the landscape of software development, testing, and verification. Unlike traditional software systems, where deterministic outputs are the norm, LLMs and LLMBAs introduce a realm of nondeterministic and generative behaviors that challenge conventional software engineering paradigms for evaluation. 

This shift is not merely a technical evolution but a fundamental transformation in how we conceive, build, and assess business applications. For that reason, remaining entrenched in traditional testing frameworks that fail to account for the probabilistic nature of LLMs and LLMBAs will inevitably lead to significant risks.

This chapter explores techniques for bridging the “evaluation gap” between traditional software testing approaches and the unique requirements of LLMs and LLMBAs and proceeds in the following way.

First, we discuss the nondeterministic nature of LLMs, how that flows through to LLMBAs, and how we can observe it. From there, we dive into how to build our own evaluation frameworks for LLMBAs. That starts with designing our framework in a thoughtful way and then proceeds to two options. The first option is to use objective metrics for evaluating LLMBAs, and the second option is to use another LLM to judge the ...