Skip to Content
View all events

Designing AI Agent Architectures

Published by O'Reilly Media, Inc.

Intermediate content levelIntermediate

Managing orchestration, memory, and evaluation

What you’ll learn and how you can apply it

  • Understand the components of an LLM-driven agentic system
  • Navigate trade-offs involved in building agents using a systematic approach
  • Define the right criteria and metrics for evaluating an agentic system based on your use case
  • Design agent architectures for your organization/use case
  • Implement a lightweight agentic framework that can form the basis for building agent applications
  • Build production-ready agentic systems

Course description

Although 2025 has been touted as the “year of the agents,” their promise hasn’t fully materialized. The technology is far from mature, with best practices still crystalizing. In this course, Suhas Pai helps you navigate the inherent uncertainties and trade-offs of this new paradigm to build robust, production-grade agents.

Suhas takes you through the components of a typical agentic system, including models, tools, memory, agent loop prompts, orchestration software, guardrails, and verifiers, and you’ll learn how to select or implement each component based on the needs of a given use case. You’ll also explore various architectural patterns that arrange these components to form a robust system. At each juncture, you’ll consider the available choices as well as the trade-offs, creating a systematic approach to implementing each component of an agent. Finally, you’ll cap off what you’ve learned by building an in-house agentic framework from which you can produce a variety of agentic applications.

This live event is for you because...

  • You’re a software engineer or architect who’s interested in building LLM-driven agents.
  • You’re a technical product manager who wants to understand the capabilities of agents and their range of possibilities.
  • You’re a machine learning scientist who wants to build agents that work.

Prerequisites

  • Python programming experience
  • Experience interacting with LLMs and accessing them through an API/Hugging Face
  • An understanding of embeddings and basic retrieval techniques

Recommended preparation:

  • Open GitHub repository (link to come)

Recommended follow-up:

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Defining agents and autonomy (60 minutes)

  • Presentation: Defining agents and autonomy; components of a typical agentic system (models, tools, memory, agent prompt loop, orchestration software, guardrails and verifiers); canonical architecture of an agent
  • Group discussion: How can we implement agents with various levels of autonomy?
  • Hands-on exercise: Build a 101 agent using an existing framework like LangGraph
  • Q&A

Components and evaluation (60 minutes)

  • Presentation: Deep dive into components of an agentic system; understanding trade-offs in implementing them; choosing the right models; integrating tools; implementing tool selection; state management; adding guardrails, verifiers, and human-in-the-loop
  • Group discussion: How to evaluate agents?; What kind of verifiers can we implement?
  • Hands-on exercise: For a given agent, implement an evaluation harness
  • Q&A

Memory and retrieval (60 minutes)

  • Presentation: Making use of external memory; memory hierarchy management; retrieval techniques
  • Group discussion: How can we ensure the model has the right context in the context window at any moment in an agentic workflow?
  • Hands-on exercise: Build a module to implement changes in context at every step of an agentic workflow
  • Q&A

Design patterns (60 minutes)

  • Presentation: Design patterns—directed acyclic graphs; behavior trees; multi-agent systems; cascades; routers; building your own agentic framework
  • Group discussion: How do we architect a deep-research agent?
  • Hands-on exercise: Build a lightweight agentic framework from scratch
  • Q&A

Your Instructor

  • Suhas Pai

    Suhas Pai is an NLP researcher and cofounder/CTO at Hudson Labs, a Toronto-based startup. He’s writing the book Designing Large Language Model Applications, now in early release for O’Reilly Media. Suhas has led and contributed to various open source models, including as co-lead of the privacy working group at BigScience, part of the BLOOM open source LLM project. He’s also active in the ML community as chair of the Toronto Machine Learning Summit conference since 2021 and NLP lead at Aggregate Intellect.

Skill covered

Software Architecture