book

System Design for the LLM Era

Name: System Design for the LLM Era
Author: Sampriti Mitra
ISBN: 9781807789930

by Sampriti Mitra

June 2026

Intermediate

272 pages

6h 27m

English

Packt Publishing

Read now

Unlock full access

Preface
Free benefits with your book
Chapter 1: Atomic Units of LLM Systems
The path to language modelsLLMs analyzedTokensEmbeddingsPre-trainingFine-tuningPrompt engineering basicsContext windowReasoningPrompt engineering strategiesLLMs for code understandingVector searchText autocompletion vs.code completionsAbstract syntax trees (ASTs)How the AST looks for a large codebaseKnowledge graphsHandling LLM trade-offs and choicesDeterministic vs.stochastic outputsRAG approaches to information retrievalData cleaningVectorizationVector searchWhy is naive RAG not always sufficient?Why do we need a knowledge graph?What is GraphRAG?Context engineeringAgentic AIModalityData residency and compliancePerformance benchmarkingKey metrics to measure:Handling failureSummary
Chapter 2: Core Architectural Patterns for LLM System Design
Designing for resilience and reliabilityPattern: the GenAI service or LLM gatewayPattern: circuit breakers with tiered fallbacksDesigning for low latencyPattern: hybrid processing (synchronous vs.asynchronous)Pattern: response streamingPattern: caching strategiesLevel 1: exact matchLevel 2: semantic matchLevel 3: proactive cachingPattern: coalesce cachingDesigning for cost optimizationPattern: the model routerPattern: dynamic traffic control (utilization-based routing)Pattern: prompt engineering and compressionDesigning for grounding and data managementPattern: retrieval-augmented generation (RAG)Pattern: the ingestion pipelinePattern: hybrid RAGPattern: function calling (tool usage)Designing for testability and observabilityPattern: golden datasetsPattern: LLM-as-a-JudgePattern: new observability metricsDesigning for security and trustPattern: mitigating prompt injectionMitigationPattern: secure output handlingMitigationPattern: mitigating excessive agencyMitigationPattern: mitigating sensitive information disclosureMitigation (data hygiene)Pattern: preventing model denial of service (MDoS)MitigationPattern: preventing data poisoningPattern: mitigating supply chain and insecure plugin vulnerabilitiesMitigationEngineering for productionTraining with test dataSourcing datasetsTesting and training for code completeTesting for chat responsesTesting for agentic workflow and tasksPattern: quantitative evaluation testingMutation testingNegative prompt testingRespecting user privacySummary
Chapter 3: Case Study: Designing AI-Native IDEs
Introduction: what are AI-driven IDEs?Functional requirementsNon-functional requirementsScale estimatesAPI designAuthenticationInitial codebase chunkingFetch server Merkle treeCode completeChat modeAgent tasksThe blueprint: high-level designClient context engineAPI gateway layerOrchestrator engineBackground agentsData storage layerEmbeddings and vector search engineSupport for context awareness of the codebaseCall graph analysisSyncing codebase changesHow often does the Merkle tree sync trigger?How does the Merkle sync flow work?Supporting code completesSupporting chat modeDatabase selection and data modellingCode chunk embeddingsUsers and user preferencesLLM modelsChat sessionsChat messagesAgent tasksPostgreSQL vs.turbopufferDeep dive into designReducing latencyCaching strategiesScalable architectureOptimized datastores:Efficient change detectionRace-to-responseSemantic caching for code explanationsImproving reliability and availability at high loadStrict timeouts for callsCircuit breaker patternRetrying with backupsBackup model strategy:Synchronous/asynchronous hybrid processing modelQueuing and async processing for chat and agentic tasksTask queuingAsync response deliveryRate limiting and throttlingImproving accuracyAdvanced prompt engineeringDynamic feedback loopMulti-model useCompiler-as-a-judgeApplying privacy patternsEphemeral guaranteeCode cannot be reconstructedMinimizing attack surfaceMiddleware PII scrubbingTraining with test dataSourcing datasetsTesting and training for code completeTesting for chat responsesTesting for agentic workflow and tasksMonitoring and user feedbackSummaryReferences
Chapter 4: Case Study: Adaptive Learning Platform
Functional requirementsNon-functional requirementsScale estimatesAPI designAdmin lesson generation and review APIsGenerate questions (sync)Generate questions (async)Retrieve questions statusRetrieve questionsReview questionBulk edit statusEdit questionCurator APIsGenerate a new lesson session APIThe blueprint: high-level designSystem I: the offline content pipelineSeed generationAPI gatewayLesson generation serviceOrchestrator engineHuman-in-the-loop reviewIngestion into the question bank databaseSystem II: the online serving pathUser request and API gatewayLesson curator serviceFetching user contextProactive curationProactive lesson curationWarm path: instant lesson deliveryCold path: handling new and returning usersOrchestrator engineModel routerPrompt engineering layerResilience layerSafety and validation layerData lake for analysisDatabase selection and data modellingUser progress dataLesson historySkill strengthCurated lessons cacheAdmin review lessons dataLessons dataDeep dive into designAsynchronous processing for content generationScalingService scaling strategyData tier scaling and cachingDependency scalingFunction calling for deterministic gradingTesting scenariosPerformance and load testingResilience and chaos testingKill a service instanceSimulate network latencyDegrade the LLMAI quality and regression testingMonitoring and alertingSummaryReferences
Chapter 5: Case Study: AI-Powered Search for E-Commerce Platforms
Functional requirementsNon-functional requirements (NFRs)Scale estimatesData storageAPI designSearch results endpointBackgroundBlueprint high-level designOffline query processing pipelineLog aggregationParallelized enrichmentQuery segmentation and classification serviceLinking and query expansion serviceRanking and indexing workerMaintaining freshnessMemory efficiencyReal-time search flowAPI gatewaySearch serviceGenAI service (LLM gateway)LLM providersProduct discovery flowOffline flowOnline flowData modellingProduct catalogproduct_catalogSmart search cachesmart_query cacheProduct search index (for keyword search and filtering)product_search_indexVector database (for semantic search)semantic_search_indexProduct discovery cacheproduct_discoveryAnalytic logs store (datalake)User profileuser_profileSystem design deep diveHow often should the offline pipeline run?Near-real-time pathHandling cachingCache tiers: hybrid strategy for speed and scaleL1 and L2 caches (K-V lookups for hot queries)Warm tier (semantic cache for the long tail)Cache warmingCache invalidationLatency and the P99 trade-offUtility vs.latencyOptimized vector search: ANNHNSW using graph-based partitioningInverted partitioningEfficient rankingOffline static ranking (global relevance)Online personalized re-rankingEnsuring robustness: availability, scalability, and resilienceScalabilityManaged shardingManaged throughputVector database scalingResilience and fault toleranceLoad balancingDatabase failoverTurbopuffer (serverless vector database)Circuit breakersMonitoringAlertingTesting and evaluationLLM as a judgeImplementationGolden set regression suiteA/B testingImplementationModel routing and fallback logicRouting strategyImplementationHuman evaluationHuman-in-the-loop for NRT trendsSummaryReferences
Chapter 6: Case Study: AI-Powered Customer Support Agent
The standard RAG patternWhy is naive RAG not enough in this case?Utilizing a knowledge graphFunctional requirementsNon-functional requirementsScale estimatesInput metricsSource document volumeVector database volumeHigh level designAPI designReal time source webhooksUser chats with customer support via Slack/WebEmbed chunked documents (internal API)System design blueprintUser query flowChat service and state managementRAG orchestrator serviceTiered orchestration and intent routingHybrid RAG approachIngestion pipelineAsynchronous triggering and elastic scaleData ingestion and chunking (Spark)Entity extraction and embedding processorVector and graph persistenceGenAI serviceOpenAI text-embedding-3-large and text-embedding-ada-002 modelsData modellingticketsmessagesBlob storage for chunked dataStorage choice and durabilityAccess patterns and loadS3 bucket pathVector databaseTechnology choiceWorkloadknowledge_chunksKnowledge graphTechnology choiceWorkloadDeep diveScalabilitySpark ingestion jobsOpenSearch Serverless auto-scalingS3Graph databaseGenAI serviceReducing latencyRetrieval optimizationSLMs for pre-processingResponse and prompt cachingAccuracy - avoiding hallucinationsExplicit prompt guardrailsValidation and evaluation suiteGolden dataset evaluationLLM-as-a-JudgeHuman-in-the-loopSmart chunking and metadataReliabilityPrivacyIngestion isolationQuery-time filteringMiddleware PII maskingMonitoring and alertingIngestion pipeline metrics (freshness and cost control)Kafka raw_documents topicIngestion slowdownEnd-to-end ingestion latencyVectorization latencyQuery path metrics (user experience and performance)End-to-end RAG latencyRetrieval latency (total)LLM inference latencyFinancial and quality metricsLLM token usage and costAccuracy / grounding scoreEscalation rateSystem HealthResource utilizationError rateDependency healthFailure mode analysis (FMA)SummaryReferences
Chapter 7: Glossary
ACDEFGHIKLMNRSTV
Chapter 8: Unlock Your Exclusive Benefits
Unlock this Book's Free Benefits in 3 Easy Steps
Index

Content preview from System Design for the LLM Era

Index

A/B testing181

AI-driven IDEs53

API design56 – 59

database selection and data modelling74 – 82

design82, 83

functional requirements54

high-level design, blueprint60

monitoring and user feedback98

non-functional requirements54, 55

scale estimates55, 56

training, with test data97

AI-driven IDEs, design

accuracy, improving95, 96

asynchronous hybrid processing model93, 94

latency, reducing83 – 90

privacy patterns, applying96, 97

reliability and availability, improving at high load90, 91

synchronous hybrid processing model93, 94

AI-driven IDEs, high-level design

API gateway layer62

background agents63

call graph analysis68

client context engine61

codebase changes, syncing68 – 70

context awareness, of codebase63 – 67

data storage layer63 ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781807789930

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

System Design for the LLM Era

by Sampriti Mitra

Index

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.