AI Engineering Intensive

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Week 1: Context Engineering, Prompt Management, and Advanced RAG (4 hours)

Segment 1: The AI Engineering Stack and Context Engineering (30 minutes)

• What AI engineering is and how it differs from traditional ML engineering
• The modern AI stack: models, orchestration, evaluation, deployment, and monitoring
• Context engineering as a core discipline: getting the right information to the model at the right time
• Choosing models: open vs. closed, cost vs. quality tradeoffs, when to use what

Segment 2: Prompt Engineering and Prompt Management (40 minutes)

• System prompts, user prompts, and tool prompts: the hierarchy and how they interact -Structured outputs: JSON mode, function calling schemas, and constrained decoding
• Prompt management as an operational discipline: versioning, A/B testing, registries, and rollback strategies
• Exercise: Build a prompt evaluation harness that scores prompt variants across multiple dimensions

Q&A + Break (10 minutes)

Segment 3: Advanced RAG: Multi-Hop Retrieval and Query Decomposition (45 minutes)

• Where simple RAG breaks down and why advanced retrieval patterns matter
• Query planning and decomposition strategies for complex questions
• Multi-hop retrieval: chaining retrievals across documents and sources
• Exercise: Build a multi-hop RAG pipeline that answers questions requiring synthesis across multiple documents

Segment 4: Hybrid Search, Re-Ranking, and Context Optimization (45 minutes)

• Combining dense and sparse retrieval for hybrid search
• Cross-encoder re-ranking, generative grading, and metadata filtering strategies
• Context window management: budgeting tokens across system prompt, retrieved docs, and conversation history
• Exercise: Add hybrid search and re-ranking to an existing RAG pipeline and measure retrieval quality improvement

Q&A + Break (10 minutes)

Segment 5: Agentic RAG and Self-Correcting Retrieval (45 minutes)

• From static retrieval to agents that decide when, what, and how to retrieve
• Self-correcting RAG: grade retrieved documents, re-retrieve on low confidence, fallback to web search
• Exercise: Build an agentic RAG system with grading, dynamic strategy selection, and self-correction

Week 1 Wrap-up and Q&A (15 minutes)

Week 2: Agents, Tools, Fine-Tuning, and Multi-Agent Systems (4 hours)

Segment 1: Agent Architectures and Design Patterns (35 minutes)

• The spectrum: prompts to chains to workflows to agents to multi-agent systems
• ReAct, reasoning, plan-and-execute, and reflection patterns and when each applies
• Why agents fail: common failure modes, the workflow-first principle, and cost tradeoffs

Segment 2: Tool Integration: MCP, CLI Agents, and Function Calling (45 minutes)

• Function calling architecture: how tool use works under the hood
• MCP: building and connecting servers for structured tool orchestration
• CLI-based coding agents: architecture, trade-offs, and when they outperform MCP-based agents
• Exercise: Build an agent with both MCP tools and native function calling and compare the approaches

Q&A + Break (10 minutes)

Segment 3: Multi-Agent Orchestration and Memory (45 minutes)

• Supervisor, hierarchical, and peer-to-peer topologies for multi-agent coordination
• Long-term memory: episodic, semantic, and procedural memory architectures for agents
• Checkpointing, recovery, and resumable workflows with LangGraph
• Exercise: Build a multi-agent system with a supervisor, shared state, and persistent memory

Segment 4: Fine-Tuning and Dataset Engineering for AI Engineers (45 minutes)

• When to fine-tune vs. prompt vs. RAG: the decision framework
• LoRA and QLoRA: parameter-efficient fine-tuning in practice
• Dataset engineering: building training data from production logs and synthetic generation
• Distillation: training smaller models from larger ones for cost and latency optimization
• Exercise: Fine-tune a small model on a domain-specific task using LoRA and compare against prompt engineering

Q&A + Break (10 minutes)

Segment 5: Multi-Agent Case Study (35 minutes)

• End-to-end walkthrough: multi-agent AI SDR system with MCP, memory, and RAG
• Architectural decisions, trade-offs, and lessons from production deployment
• Exercise: Extend the case study system with a new agent role and tool integration

Week 2 Wrap-up and Q&A (15 minutes)

Week 3: Evaluation, Observability, and Production (4 hours)

Segment 1: Agent Harnesses, Benchmarking, and Evaluation Pipelines (45 minutes)

• Why agent evaluation is fundamentally different from model evaluation
• Agent harnesses: comparing frameworks for running, testing, and scoring agent behavior in controlled environments
• Benchmarking agents on SWE-bench, HumanEval, GAIA, WebArena, and domain-specific evaluation suites
• Building rubrics and automated evaluators: LLM-as-judge, trajectory scoring, and regression testing
• Exercise: Build an agent harness that benchmarks a RAG agent and scores tool selection, retrieval, and output quality

Segment 2: Observability, Traceability, and Debugging (40 minutes)

• Tracing agent execution end-to-end: from user query through retrieval, model calls, tool use, and response
• Structured observability with LangSmith, OpenTelemetry, and custom logging
• Debugging multi-step failures: identifying where and why agents go wrong
• Traceability for compliance: audit trails, reproducibility, and lineage tracking
• Exercise: Add tracing and structured logging to an existing agent and diagnose a deliberately planted failure

Q&A + Break (10 minutes)

Segment 3: Monitoring, Cost Control, and Inference Optimization (45 minutes)

• Production monitoring: quality degradation detection, drift alerting, and feedback loops
• Cost management: token tracking per node, model routing for cost optimization, prompt caching strategies
• Latency optimization: batching, KV caching, and when to use quantized models
• Exercise: Implement a model router that selects between a cheap and expensive model based on query complexity

Segment 4: Human-in-the-Loop, Guardrails, and Prompt Ops (40 minutes)

• Designing human review points for high-stakes agent decisions
• Input and output guardrails, compliance checks, and safety boundaries
• Prompt ops in production: versioning, A/B testing in production, tracking prompt performance over time
• Exercise: Add human-in-the-loop approval, guardrails, and prompt version tracking to an agent workflow

Q&A + Break (10 minutes)

Segment 5: Deployment and Production Case Studies (35 minutes)

• Packaging agent systems for deployment: APIs, containers, and LangGraph Platform
• Case study: production deep research workflow with multi-agent coordination, RAG, and memory
• Case study: self-improving database agent with evaluation-driven iteration
• Architectural decision framework: choosing the right pattern for your use case

**Course Wrap-up and Final Q&A (15 minutes) **