book

LLM Design Patterns

Name: LLM Design Patterns
Author: Ken Huang
ISBN: 9781836207030

by Ken Huang

May 2025

Intermediate to advanced

538 pages

13h 11m

English

Packt Publishing

Read now

Unlock full access

Designing LLM Patterns
Contributors
About the author
About the reviewers
Join our Discord and Reddit space
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the color imagesConventions usedGet in touchShare Your ThoughtsFree Benefits with Your Book
Part 1: Introduction and Data Preparation
Chapter 1: Introduction to LLM Design Patterns
Understanding LLMsThe evolution of language modelsCore features of LLMsUnderstanding design patternsOrigins and evolutionCore principles of design patternsDesign patterns for LLM developmentBenefits of LLM design patternsChallenges in applying design patterns to LLMsSummary
Chapter 2: Data Cleaning for LLM Training
Understanding the importance of clean dataCommon data quality issues in language datasetsText preprocessing techniques for LLMsHandling multilingual and code-mixed dataDeduplication strategies for large text corporaExact match deduplicationNear-duplicate detectionShinglingLocality Sensitive Hashing (LSH)Automated data cleaning pipelinesData validation and quality assuranceSummarySubscribe for a free eBook
Chapter 3: Data Augmentation
Text data augmentation techniquesSynonym replacementBack-translationText generation with T5Leveraging existing LLMs for data generationMultilingual data augmentation strategiesCross-lingual back-translationMultilingual T5 augmentationSemantic preservation in text augmentationUse of sentence embeddingsContextual word embeddings for synonym replacementBalancing augmentation and data qualityQuality filteringHuman-in-the-loop validationEvaluating the impact of data augmentationPerplexityTask-specific metricsDiversity metricsSummary
Chapter 4: Handling Large Datasets for LLM Training
Challenges of large datasetsData sampling techniquesDistributed data processingData sharding and parallelization strategiesEfficient data storage formatsStreaming data processing for continuous LLM trainingMemory-efficient data loading techniquesSummarySubscribe for a free eBook

Chapter 5: Data Versioning
Understanding the need for data versioningData versioning strategies for large language datasetsTools for data versioningIntegrating data versioning in training workflowsVersion control for text corporaManaging dataset variants and experimentsBest practices for data versioningSummary
Chapter 6: Dataset Annotation and Labeling
The importance of quality annotationsAnnotation strategies for different tasksTools and platforms for large-scale text annotationManaging annotation qualityCrowdsourcing annotations – benefits and challengesSemi-automated annotation techniquesScaling annotation processes for massive language datasetsAnnotation biases and mitigation strategiesSummarySubscribe for a free eBook
Part 2: Training and Optimization of Large Language Models
Chapter 7: Training Pipeline
Components of a training pipelineData input and preprocessingLLM architecture design considerationsLoss functions and optimization strategiesLoggingPipeline modularity and reusabilityScaling your training pipeline for larger modelsSummary
Chapter 8: Hyperparameter Tuning
Understanding hyperparametersManual versus automated tuningManual tuningAutomated tuningGrid and random searchBayesian optimizationPopulation-based methodsMulti-objective hyperparameter optimizationHyperparameter tuning at scale – challenges and solutionsSummarySubscribe for a free eBook
Chapter 9: Regularization
L2 regularization (Ridge regression)DropoutLayer-wise adaptive regularizationGradient clipping and noise injectionRegularization in transfer learning and fine-tuning scenariosEmerging regularization techniquesStochastic weight averagingSharpness-aware minimizationDifferential privacy-based regularizationFast gradient sign methodLookahead optimizerSummary
Chapter 10: Checkpointing and Recovery
Why is checkpointing important?Checkpoint frequency and storage strategiesEfficient checkpoint formatsRecovering from failuresCheckpointing in distributed LLM trainingVersion control for LLM checkpointsAutomated checkpointing and recovery systemsSummarySubscribe for a free eBook
Chapter 11: Fine-Tuning
Implementing transfer learning and fine-tuningStrategies for freezing and unfreezing layersLearning rate schedulingDomain-specific fine-tuning techniquesFew-shot and zero-shot fine-tuningContinual fine-tuning and catastrophic forgettingSummary
Chapter 12: Model Pruning
Magnitude-based pruningStructured versus unstructured pruningIterative pruning techniquesPruning during training versus post-training pruningBalancing pruning and model performanceCombining pruning with other compression techniquesPruning and quantizationPruning and knowledge distillationSummarySubscribe for a free eBook
Chapter 13: Quantization
Understanding the basicsPTQQATMixed-precision quantizationHardware-specific considerationsComparing quantization strategiesCombining quantization with other optimization techniquesPruning and quantizationKnowledge distillation and quantizationSummary
Part 3: Evaluation and Interpretation of Large Language Models
Chapter 14: Evaluation Metrics
NLU benchmarksMassive multitask language understandingSuperGLUETruthfulQAReasoning and problem-solving metricsAI2 Reasoning ChallengeGrade School Math 8KCoding and programming evaluationConversational ability assessmentCommonsense and general knowledge benchmarksOther commonly used benchmarksDeveloping custom metrics and benchmarksInterpreting and comparing LLM evaluation resultsSummarySubscribe for a free eBook
Chapter 15: Cross-Validation
Pre-training and fine-tuning data splitsStratified sampling for pre-training dataTime-based splitting for fine-tuning dataOversampling and weighting techniques for data balancingFew-shot and zero-shot evaluation strategiesFew-shot evaluationZero-shot evaluationDomain and task generalizationEvaluating domain adaptationEvaluating task generalizationContinual learning evaluationCross-validation challenges and best practicesSummary
Chapter 16: Interpretability
Attention visualization techniquesProbing methodsExplaining LLM predictions with attribution methodsInterpretability in transformer-based LLMsMechanistic interpretabilityTrade-offs between interpretability and performanceSummarySubscribe for a free eBook
Chapter 17: Fairness and Bias Detection
Types of biasFairness metrics for LLM text generation and understandingDetecting biasDebiasing strategiesFairness-aware trainingEthical considerationsSummary
Chapter 18: Adversarial Robustness
Types of textual adversarial attacksAdversarial training techniquesEvaluating robustnessTrade-offs in the adversarial training of LLMsReal-world implicationsSummarySubscribe for a free eBook
Chapter 19: Reinforcement Learning from Human Feedback
Components of RLHF systemsReward modelPolicy optimizationScaling RLHFLimitations of RLHF in language modelingApplications of RLHFSummary
Part 4: Advanced Prompt Engineering Techniques
Chapter 20: Chain-of-Thought Prompting
Designing effective CoT promptsUsing CoT prompting for problem solvingCombining CoT prompting with other techniquesEvaluating CoT prompting outputsLimitations of CoT promptingFuture directionsSummarySubscribe for a free eBook
Chapter 21: Tree-of-Thoughts Prompting
Designing ToT promptsSearch strategiesPruning and evaluationApplying ToT to solve a multi-step problemChallenges in implementationFuture directionsSummary
Chapter 22: Reasoning and Acting
Implementing ReAct in LangChainReAct Document StoreBuilding ReAct agents with LangChain’s Expression LanguageExplaining ReActSingleInputOutputParserRunning the agent with AgentExecutorCompleting tasks and solving problemsEvaluating ReAct’s performanceSafety, control, and ethical considerationsLimitations and future directionsSummarySubscribe for a free eBook
Chapter 23: Reasoning WithOut Observation
Implementing ReWOO with LangGraphAdvantages of ReWOOEvaluating quality and ethical considerationsFuture directionsSummary
Chapter 24: Reflection Techniques
Designing prompts for self-reflectionImplementing iterative refinementCorrecting errorsEvaluating the impact of reflectionChallenges in implementing effective reflectionFuture directionsSummarySubscribe for a free eBook
Chapter 25: Automatic Multi-Step Reasoning and Tool Use
Designing prompts for complex task decompositionIntegrating external toolsImplementing automatic tool selection and useComplex problem solvingEvaluating multi-step reasoning and tool useChallenges and future directionsSummary
Part 5: Retrieval and Knowledge Integration in Large Language Models
Chapter 26: Retrieval-Augmented Generation
Building a simple RAG system for LLMsEmbeddings and indexing for retrieval in LLM applicationsEmbeddingsIndexingExample code demonstrating embedding, indexing, and searchingQuery formulation strategies in LLM-based RAGIntegrating retrieved information with LLM generationChallenges and opportunities in RAG for LLMsSummarySubscribe for a free eBook
Chapter 27: Graph-Based RAG
Introduction to graph-based knowledge representation for LLMsDesigning graph RAG architectures for LLMsGraph embedding techniques for LLM retrievalQuery expansion using graph structures in LLMsApplications and use cases of graph RAG in LLMsChallenges and future directions in graph-based RAGSummary
Chapter 28: Advanced RAG
Multi-step and iterative retrieval techniques for LLMsAdaptive retrieval based on context and task in LLMsMeta-learning for improved retrieval in LLMsCombining RAG with other LLM prompting techniquesHandling ambiguity and uncertainty in LLM-based RAGScaling RAG to very large knowledge basesFuture directions in RAG research for LLMsSummarySubscribe for a free eBook
Chapter 29: Evaluating RAG Systems
Challenges in evaluating RAG systems for LLMsThe interplay between retrieval and generationContext-sensitive evaluationBeyond factual accuracyLimitations of automated metricsDifficulty in error analysisThe need for diverse evaluation scenariosDynamic knowledge and evolving informationComputational costMetrics for assessing retrieval quality in LLM-based RAGRecall@kPrecision@kMean Reciprocal Rank (MRR)Normalized Discounted Cumulative Gain (NDCG@k)Considerations for retrieval metrics in RAGEvaluating the relevance of retrieved information for LLMsMethods for evaluating the relevance of retrieved informationChallenges in evaluating RAG-specific relevanceMeasuring the impact of retrieval on LLM generationKey metrics for evaluating retrieval impactChallenges in measuring the impact of retrievalEnd-to-end evaluation of RAG systems in LLMsEvaluation strategiesChallenges in end-to-end evaluationHuman evaluation techniques for LLM-based RAGBest practices for human evaluationChallenges in human evaluationBenchmarks and datasets for RAG evaluationSummary
Chapter 30: Agentic Patterns
Introduction to agentic AI systems based on LLMsGoal-setting and planning in LLM-based agentsImplementing memory and state management for LLM agentsDecision-making and action selection in LLM-based agentsLearning and adaptation in agentic LLM systemsEthical considerations and safety in LLM-based agentic AIFuture prospects of agentic AI using LLMsSummaryFuture directions in LLM patterns and their developmentSubscribe for a free eBook
Chapter 31: Unlock Your Exclusive Benefits
Unlock this Book’s Free Benefits in 3 Easy StepsStep 1
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from LLM Design Patterns

19 Reinforcement Learning from Human Feedback

In this chapter, we’ll dive into Reinforcement Learning from Human Feedback (RLHF), a powerful technique for aligning LLMs with human preferences. RLHF combines reinforcement learning with human feedback to fine-tune language models. It aims to align the model’s outputs with human preferences, improving the quality and safety of generated text.

RLHF differs from standard supervised fine-tuning by optimizing for human preferences rather than predefined correct answers. While supervised learning minimizes loss against labeled examples, RLHF creates a reward model from human comparisons between model outputs and then uses this reward function (typically with proximal policy optimization (PPO)) to update ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781836207030

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

LLM Design Patterns

by Ken Huang

19

Reinforcement Learning from Human Feedback

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.