Building AI Agent Platforms

Chapter 7. Experimentation

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 7th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

“It doesn’t matter how beautiful your theory is…​ If it doesn’t agree with experiment, it’s wrong.”

Richard Feynman

The major difference between developing traditional software and AI applications is that the latter rarely follows a straight line to production. Progress frequently comes very quickly at first, then stagnates or even regresses. This is due to the stochastic and unpredictable nature of AI models, which makes it difficult to predict which changes will lead to improvements.

Does changing the prompt lead to better performance? Does switching to a different model improve the quality of responses? Which retrieval mechanism best supports factual accuracy? While there are proven strategies to follow, ultimately these questions cannot be answered with certainty a priori. Instead, teams must test them and measure the results. Consequently, it is effectively impossible to plot the shortest path to production.

In contrast, traditional software development exhibits ...