Generative AI Design Patterns

Chapter 6. Improving Reliability

Foundational models are inherently stochastic, which means they involve random variables or probability. As a result, GenAI applications built on top of these models can suffer from inconsistent output, factual inaccuracies, and hallucinations. This chapter introduces four patterns that are designed to mitigate these challenges.

LLM-as-Judge (Pattern 17) facilitates systematic evaluation of output by employing LLMs, either directly via prompting or through ML/fine-tuned models, to assess quality. Reflection (Pattern 18) enables models to critique their own work and iteratively refine generated content. Dependency Injection (Pattern 19) promotes testability and robustness by enabling the mocking of components. Finally, Prompt Optimization (Pattern 20) offers a structured approach to refining input prompts and thus maximizes output reliability across diverse input distributions.

Together, these patterns form a framework that enhances the dependability of GenAI applications. By integrating mechanisms for evaluation, self-correction, uncertainty quantification, modular design, and input optimization, you can build more reliable systems and foster greater user confidence in the outputs generated by your AI applications.

Pattern 17: LLM-as-Judge

The LLM-as-Judge pattern is a way to provide detailed, multidimensional feedback that you can use to compare models, track improvements, and guide further development. LLM-as-Judge represents a promising middle ...