Large Language Models: The Hard Parts

Chapter 3. Evaluation Tools for LLM-Based Applications

We shape our tools and thereafter our tools shape us.

Fr. John Culkin (often attributed to Marshall McLuhan)

Let’s move on to three tools that help us scale our evaluation work: LangSmith, Promptfoo, and LightEval. LangSmith makes it easy to scale and monitor the pipelines we covered in Chapter 2. Promptfoo takes that concept of scale and makes it simple to apply across the world of prompting. LightEval, developed by Hugging Face, takes us into benchmark-style model evaluation across standardized tasks. By the end of the chapter, we will have a good grasp on scaling model, prompt, and benchmark evaluations. Of these three tools, Promptfoo and LightEval are open source, while LangSmith is not. Although this book prioritizes open source tools, we include LangSmith because it is widely used in practice and provides a useful point of comparison with open source alternatives.

LangSmith

LangSmith is a developer framework that provides comprehensive tools for building, testing, monitoring, and evaluating LLMs and LLMBAs. It enables teams to track model performance, identify issues in production, and continuously improve LLMBAs through systematic evaluation against defined criteria. LangSmith was built and is developed and maintained by the same team that builds LangChain, a popular library for LLM development.

To learn more about using LangSmith, let’s revisit our previous example of an LLMBA that generates a short summary of ...