AI Evaluations Bootcamp
Published by O'Reilly Media, Inc.
A systematic approach to understanding what works and what doesn’t in modern AI systems
What you’ll learn and how you can apply it
- Frame and scope evaluation goals for modern AI applications, including LLM, RAG, and agentic AI systems, across dimensions such as effectiveness, robustness, and safety
- Choose and apply appropriate evaluation methods, including automated metrics, human judgments, and LLM-as-a-judge techniques
- Design and build meaningful evaluation datasets, including synthetic and adversarial inputs
- Evaluate the performance of RAG and agentic RAG systems using tools such as RAGAS and DeepEval.
- Perform robustness testing by generating and using adversarial evaluation data. using tools such as RAGAS and DeepEval
- Interpret and communicate evaluation results effectively, avoiding common pitfalls and tailoring insights to different stakeholders
As large language models (LLMs), retrieval-augmented generation (RAG) systems, and AI agents are increasingly deployed across search, chat, decision-support, and workflow automation applications, the need for systematic and meaningful evaluation has never been greater. Traditional ML evaluation techniques often fall short when applied to open-ended, generative outputs, leaving teams without clear insight into how well their models are actually performing.
Join Panos Alexopoulos to discover a systematic and rigorous approach to evaluating AI applications. You’ll learn how to define evaluation objectives based on task requirements, select appropriate evaluation dimensions, and apply both automated and human-in-the-loop methods, including LLM-as-a-judge techniques. Through hands-on Jupyter notebooks, you’ll design synthetic and adversarial evaluation datasets, implement and evaluate LLM judges and (agentic) RAG systems, and diagnose failure modes in real-world use cases. You’ll also gain an understanding of interpretation and reporting best practices so that you can communicate results effectively and support informed deployment decisions.
This live event is for you because...
- You’re an AI/ML engineer, product owner, or auditor working on the development, evaluation, or deployment of LLM-powered systems.
- You work with LLM-based applications such as chatbots, summarizers, RAG pipelines, or prompt-based workflows and need to ensure their performance, reliability, and safety in real-world settings.
- You want to confidently evaluate and improve the performance, safety, and robustness of LLM-based applications in real-world settings.
Prerequisites
- Foundational knowledge of machine learning and LLMs
- Basic- to intermediate-level Python programming experience
- Awareness of Jupyter notebooks and Google Colab
Recommended preparation:
- Links to Google Colab notebooks will be provided prior to the course
Recommended follow-up:
- Read “Evaluation Methodology” and “Evaluate AI Systems” (chapters 3 and 4 in AI Engineering)
- Read Evals for AI Engineers (book)
Schedule
The time frames are only estimates and may vary according to how the class is progressing.
Day 1
Introduction to AI systems evaluation (45 minutes)
- Presentation: What is AI evaluation and why is it important; types of AI systems; why AI evaluation is challenging; A systematic approach to AI evaluation
- Group discussion: Which is the hardest part of AI evaluation for your team?
- Q&A
- Break
Framing and scoping the evaluation problem (45 minutes)
- Presentation: Why clear evaluation scope matters; defining evaluation goals; determining evaluation objects and boundaries; choosing evaluation dimensions; common pitfalls
- Group discussion: Which evaluation goals and dimensions are more relevant to your current project?
- Q&A
- Break
Picking evaluation metrics (45 minutes)
- Presentation: Why choosing the right metrics matters; overview of AI metrics; what makes a good evaluation metric
- Q&A
- Break
Acquiring evaluation data and judgments (105 minutes)
- Presentation: Types and sources of evaluation data; what makes good evaluation data; data collection and sampling; labeling, annotating, and judging using humans and AI; synthesizing evaluation data
- Jupyter notebook: Evaluating LLM judges
- Jupyter notebook: Create synthetic evaluation data for natural language inference
- Group discussion: When is it appropriate to use synthetic evaluation data?
- Q&A
Day 2
Deep dive: Evaluating RAG and Agentic RAG systems (120 minutes)
- Presentation: Understanding RAG, GraphRAG and agentic RAG architectures; Picking RAG evaluation metrics; creating RAG evaluation data
- Jupyter notebook: Implement and evaluate a GraphRAG system for career advice using RAGAs and DeepEval
- Jupyter notebook: Implement and evaluate an agentic GraphRAG system
- Q&A
- Break
Deep dive: Performing adversarial evaluation for robustness and safety (100 minutes)
- Presentation: Understanding AI vulnerabilities and risks; designing adversarial test cases and datasets; performing error analysis and root cause diagnosis.
- Jupyter notebook: Adversarial evaluation of LLM judges
- Jupyter notebook: Adversarial evaluation of LLM susceptibility to cognitive biases
- Q&A
- Break
Interpreting, communicating, and acting on evaluation results (20 minutes)
- Presentation: The stakes of good interpretation and communication of AI evaluation; interpretation and communication pitfalls and risks; communication strategies and techniques; translating evaluation findings into improvement actions
- Group discussion: How do you balance honesty about system limitations with stakeholder pressure to show progress?
- Q&A
Your Instructor
Panos Alexopoulos
Panos Alexopoulos works as head of ontology at Textkernel, in Amsterdam, Netherlands, where he leads a team of data professionals developing and delivering a large cross-lingual knowledge graph in the HR and recruitment domain. He’s the author of Semantic Modeling for Data: Avoiding Pitfalls and Breaking Dilemmas (O'Reilly) and has published more than 60 papers at international conferences and in journals and books. He has worked at the intersection of data, semantics, and software since 2006, contributing to building intelligent systems that deliver value to business and society. Born and raised in Athens, Greece, Panos holds a PhD in knowledge engineering and management from National Technical University of Athens and is a regular speaker and trainer in both academic and industry venues.