AI Agent Memory Management Bootcamp

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Day 1

AI agent memory (55 minutes)

• Presentation: Overview of AI memory concepts: short-term (context window) versus long-term (conversational memory), episodic versus semantic, and their differences; analogies to human cognition and memory; challenges of memory in AI agents and agentic systems; real-world examples and intuition behind AI memory
• Group discussion: Which aspect of memory do you find most challenging to conceptualize?
• Q&A
• Break

Agent memory management (65 minutes)

• Presentation: Overview of retrieval-augmented generation (RAG) methods: lexical, vector, hybrid, agentic RAG, and rerankers; core components of memory management: initialization, segmentation, structure, creation, retrieval, update, deletion; best practices and design patterns in agentic systems; overview of memory design architectures from generative agents, A-Mem, MemGPT, and agent workflow memory
• Group discussion: Which memory component do you believe is most critical for long-term agent coherence?
• Q&A
• Break

AI retrieval (60 minutes)

• Presentation: Overview of AI retrieval: definition, concepts, and techniques; introduction to the concept of "memory providers" and their role in AI application architecture; examples of memory providers
• Hands-on exercises: Setting up the environment; indexing and vector search implementation; building retrieval functions; end-to-end agentic RAG system; performance monitoring and logging implementation
• Q&A
• Break

Agent memory technique implementation (50 minutes)

• Hands-on exercises: Implementing memory management components using Oracle AI Database, LangGraph, and LangMem
• Hands-on exercises: Implementing context engineering and memory engineering techniques from the ground up: encoding strategies, context window management, and memory injection patterns
• Hands-on exercises: Understanding the difference between memory-augmented and memory-aware agents: architectural distinctions, capability boundaries, and when each pattern applies
• Hands-on exercises: Overview and implementation of agent-triggered and deterministic memory operations: when memory actions are initiated by the agent versus enforced by the system, and the trade-offs between the two

Wrap-up and Q&A (10 minutes)

Day 2

Traditional evaluation metrics and RAG pipeline metrics (55 minutes)

• Presentation: Anatomy of a memory-aware agent and the case for evaluation: memory store, ingestion pipeline, retrieval layer, context assembly layer, and agent reasoning loop, each mapped to its failure modes and evaluation targets; classical IR foundations: Precision, Recall, and F1; ranking quality metrics: MRR, nDCG, and MAP; RAG-specific evaluation dimensions: Faithfulness, Answer Relevance, Context Precision, and Context Recall; RAG evaluation frameworks: RAGAS and Galileo; chunking and indexing quality signals
• Notebook: Computing IR metrics from scratch in Python; RAG evaluation with RAGAS; production tracing and evaluation with Galileo
• Q&A
• Break

Agent evaluation metrics (55 minutes)

• Presentation: Why agent evaluation differs from RAG evaluation; task completion and goal achievement: binary and partial-credit scoring; tool use accuracy: precision and recall applied to tool invocation sequences; trajectory and decision quality: trajectory faithfulness metrics; latency, efficiency, and cost metrics; agent evaluation frameworks: AgentBench, AgentEval, and LangSmith; safety and alignment metrics: adversarial test sets and red-teaming
• Notebook: Scoring task completion; tool use evaluation; trajectory evaluation with LangSmith; efficiency and cost profiling
• Q&A
• Break

Memory evaluation as a distinct discipline (MemEval) (55 minutes)

• Presentation: Memory-specific failure modes not captured by RAG or agent metrics; unit-level memory evaluation: precision and recall at the memory-block level; memory retrieval evaluation: MRR, nDCG, and Hit Rate applied to memory queries, including suppression quality; continuity and consistency evaluation: cross-session correctness, persona stability, and contradiction detection; memory lifecycle evaluation: ingestion, consolidation, forgetting, and archiving; LoCoMo, LongMemEval, and MemBench; the RBC lens for memory evaluation: Reliable, Believable, Capable
• Notebook: Evaluating memory units in practice; memory retrieval scoring; running LoCoMo, LongMemEval, and MemBench; building a continuous memory evaluation pipeline
• Q&A
• Break

Memory benchmarking and competitive analysis (65 minutes)

• Presentation: What memory benchmarking means in practice: defining comparison criteria, selecting representative workloads, and establishing baseline performance; overview of memory system architectures under comparison: filesystem-based, vector database, relational, and hybrid approaches; evaluation dimensions for competitive analysis: retrieval latency, recall quality, consistency under load, memory lifecycle support, and cost per operation; interpreting benchmark results: avoiding benchmark gaming, understanding workload sensitivity, and translating results into architecture decisions
• Notebook: Running a structured benchmark across two or more memory system configurations; scoring each system against retrieval quality, latency, and lifecycle metrics; producing a comparative analysis report with findings and recommendations
• Q&A

Wrap-up and Q&A (10 minutes)