book

Build an AI Agent (From Scratch)

Name: Build an AI Agent (From Scratch)
ISBN: 9781633434615

by jungjun Hur, Younghee Song

July 2026

Intermediate

336 pages

9h 36m

English

Manning Publications

Read now

Unlock full access

Build an AI Agent (From Scratch)
copyright
contents
preface
acknowledgements
about this book
about the authors
about the cover illustration
Part 1 Building your first LLM agent
1 What is an AI agent?
1.1 The age of AI agents1.2 Understanding LLM agents1.2.1 What is an LLM?1.2.2 What is an LLM agent?1.3 Workflow vs. agent1.3.1 Workflow: Developer-defined flow1.3.2 Agent: LLM-directed flow1.3.3 Combining workflows and agents in practice1.4 Tasks that require agents1.4.1 Tasks that require an LLM1.4.2 Conditions for using agents1.4.3 GAIA: An agent gym1.5 Context engineering1.5.1 Why agents fail1.5.2 From prompt engineering to context engineering1.5.3 Bigger context isn’t always better1.5.4 Five context-engineering strategies1.5.5 From context engineering to harness engineering1.5.6 The journey of this book1.6 Prerequisites for reading this book

2 The brain of AI agents: LLMs
2.1 Choosing LLMs for agents2.1.1 Starting with closed LLMs2.1.2 Expanding to open LLMs2.1.3 Essential LLM capabilities for agents2.2 LLM API basics for building agents2.2.1 Setting up the development environment2.2.2 Getting started with the OpenAI API2.2.3 Unifying providers with LiteLLM2.2.4 Conversation management: Handling stateless APIs2.2.5 Generating structured output2.2.6 Asynchronous calls2.3 Enhancing agent intelligence: Prompt engineering2.3.1 The role of system prompts2.3.2 Guidelines for agent prompts2.4 Experiencing LLM limitations with GAIA2.4.1 Why GAIA: Setting goals for agent development2.4.2 Experiment setup2.4.3 Results and analysis: Why LLMs need tools
3 Enabling actions: Tool use
3.1 LLM tools3.1.1 Why do we need LLM tools?3.1.2 Types of LLM tools3.2 How LLMs use tools3.2.1 How tool calling works3.2.2 How an LLM chooses tools3.2.3 Guidelines for effective tool calling3.3 Building tools and tool definitions for LLMs3.3.1 Implementing a web search tool3.3.2 Converting to tool definitions3.3.3 End-to-end tool execution3.3.4 The challenges of custom tools3.4 MCP: Standardizing tools3.4.1 The core of MCP: Server–client architecture3.4.2 Hands-on: Running an MCP server3.4.3 Understanding the MCP client3.4.4 Hands-on: Implementing an MCP server
4 Implementing a basic ReAct agent
4.1 How ReAct agents work4.1.1 The think-act cycle4.1.2 From text parsing to tool calling4.2 Agent architecture overview4.2.1 The completed agent4.2.2 Information flow: The core design4.2.3 Components we have to build4.3 ExecutionContext: The agent’s central storage4.3.1 What happens during agent execution4.3.2 ExecutionContext implementation4.4 Tool abstraction4.4.1 Why we need a unified tool interface4.4.2 BaseTool: The foundation4.4.3 FunctionTool: Wrapping functions4.4.4 Integrating MCP tools4.5 LLM communication layer4.5.1 Reasons for adding a communication layer4.5.2 LlmRequest: Selecting what to send4.5.3 LlmResponse: Standardizing what we receive4.5.4 LlmClient: The provider adapter4.5.5 Putting it together4.6 Implementing the agent4.6.1 Agent class structure4.6.2 The run() method4.6.3 The step() method4.6.4 The think() and act() methods4.7 Adding structured output4.7.1 The approach: Tools as output formatters4.7.2 Modifying the agent4.7.3 Using structured output in practice4.8 Testing with the GAIA benchmark4.8.1 From LLM to agent4.8.2 Results
Part 2 Developing advanced agent capabilities
5 Building knowledge bases with RAG
5.1 The problem of using internal data5.1.1 A simple case: Single file5.1.2 Multiple files5.1.3 Large or extensive data5.2 Types of search methods5.2.1 Keyword search5.2.2 Vector search5.2.3 Graph search5.2.4 Structure-based search5.3 Practicing vector search5.3.1 Embedding: Converting text to vectors5.3.2 Chunking: Dividing long text into search units5.3.3 Implementing vector search5.3.4 Why vector databases are necessary5.3.5 Exercise: Finding relevant information from web search results5.4 Structure-based search5.4.1 Preparing the GAIA dataset5.4.2 Implementing filesystem tools5.4.3 Connecting tools to the agent5.4.4 Solving GAIA zip file problems5.5 Extending agents with callbacks5.5.1 The need for agent extension5.5.2 Implementing tool callbacks5.5.3 Human-in-the-loop tool-execution approval5.5.4 Compressing search results
6 Adding memory to your agent
6.1 The anatomy of agent memory6.1.1 Limitations of the current memory architecture6.1.2 Three challenges of the current memory architecture6.1.3 Context engineering and memory6.2 Managing context during execution6.2.1 Separating storage from presentation6.2.2 Sliding-window strategy6.2.3 Token counting6.2.4 Compaction strategy6.2.5 Summarization strategy6.2.6 Hierarchical context management6.3 Continuous execution: Session and state management6.3.1 The Session class6.3.2 Managing sessions with SessionManager6.3.3 Integrating sessions into the agent6.3.4 Basic example: Multiturn conversation6.3.5 Data structures for tool confirmation6.3.6 Extending tools for confirmation6.3.7 Implementing pause and resume in the agent6.3.8 Complete example: HITL workflow6.4 Long-term memory: Accumulating knowledge across sessions6.4.1 The structure of long-term memory6.4.2 Information extraction: Structured output6.4.3 Building a vector store with ChromaDB6.4.4 Implementing TaskMemoryManager6.4.5 Retrieving memories
7 Planning and reflection for complex tasks
7.1 Giving agents time to think7.1.1 The limitations of ReAct7.1.2 How human experts work7.1.3 Why time to think matters7.2 Planning: Setting direction7.2.1 When planning is necessary7.2.2 Implementing the planning tool7.2.3 Planning-tool example7.2.4 Extension directions7.3 Reflection: Checking and correcting7.3.1 When reflection is necessary7.3.2 Implementing the reflection tool7.3.3 The real value of reflection: Failure recovery7.3.4 Running an agent that uses reflection for research synthesis7.4 Integrating planning and reflection
8 Empowering agents with code execution
8.1 Giving agents a computer8.1.1 Limitations of the predefined-tool approach8.1.2 What code execution brings8.1.3 The effectiveness of code-based actions8.2 Connecting code environments8.2.1 Why sandboxes are necessary8.2.2 Introduction to E2B8.2.3 Connecting the sandbox to the agent8.2.4 Testing the code-execution agent8.3 Porting tools to sandboxes8.3.1 The need for tool portability8.3.2 Extending FunctionTool8.3.3 Implementing the Wikipedia revision tool8.3.4 Handling sandbox tools in the Agent8.3.5 Practical testing8.4 Workspace: Using filesystems and the CLI8.4.1 The workspace concept8.4.2 Implementing workspace tools8.4.3 Practical example: Analyzing an Excel file8.5 Agent skills8.5.1 The agent skills concept8.5.2 Progressive tool-information disclosure8.5.3 Implementation8.5.4 Practical example: PDF merge skill8.5.5 Hierarchical tool structure
9 Orchestrating multi-agent systems
9.1 Why multi-agent?9.2 Three collaboration patterns9.3 Agent workflow: Defining order in code9.3.1 Workflow design principles9.3.2 Sequential: Step-by-step execution9.3.3 Parallel: Concurrent execution9.3.4 Loop: Iterative execution9.3.5 Composing workflows9.4 Agent as Tool: Calling agents as tools9.4.1 Making agents into tools9.4.2 Implementing AgentTool9.4.3 Context isolation9.4.4 Practical example: Researcher and coder collaboration9.5 Agent Transfer pattern9.5.1 Differences between Agent as Tool and Transfer9.5.2 Transfer is also a tool9.5.3 Agent tree9.5.4 Implementing the transfer tool9.5.5 Implementing agent transfer9.6 A2A: Collaborating across networks9.6.1 The core of A2A: Agent card and task request/response9.6.2 Server: Exposing an agent to the network9.6.3 Client: Calling remote agents
10 Evaluating agents
10.1 Observing an agent10.1.1 Pillars of observability: Metrics, traces, and logs10.1.2 Generating, collecting, and exporting telemetry: OpenTelemetry10.2 Building datasets and establishing evaluation criteria10.2.1 What should we evaluate?10.2.2 Creating a dataset10.2.3 Analyzing errors10.2.4 Designing rubrics and metrics10.3 Evaluating with LLM-as-a-judge10.3.1 Types of LLM-as-a-judge10.3.2 Building a rubric-based evaluation system10.4 Operations: CI/CD10.4.1 Evaluation-gated deployment10.4.2 Improving the test set and evaluator: Agent quality flywheel
appendix. OpenAI API Key

Content preview from Build an AI Agent (From Scratch)

1 What is an AI agent?

This chapter covers

The landscape of AI agents today
LLMs as the decision-making core of agents
Workflows versus agents and when to use each
GAIA benchmark for measuring agent performance
Context engineering for building effective agents

You may have heard of agent-building frameworks such as LangGraph, CrewAI, AutoGen, and OpenAI Agents. These frameworks make it easy to build agents quickly, but they also hide what’s happening inside. This book takes a different approach: we’ll build agents from scratch, understanding every component before relying on any framework.

Why build from scratch? Agent development is fundamentally about debugging failures. When your agent gives a wrong answer or gets stuck in a loop, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Build an AI Agent (from Scratch): Coding Video

Publisher Resources

ISBN: 9781633434615Publisher Support Other Publisher Website Purchase Link

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Build an AI Agent (From Scratch)

by jungjun Hur, Younghee Song