book

Building Large Language Models from Scratch: Design, Train, and Deploy LLMs with PyTorch

Name: Building Large Language Models from Scratch: Design, Train, and Deploy LLMs with PyTorch
Author: Dilyan Grigorov
ISBN: 9798868822971

by Dilyan Grigorov

April 2026

Intermediate

547 pages

11h 21m

English

Apress

Read now

Unlock full access

Building Large Language Models from Scratch
Introduction
Table of Contents
About the Author
About the Technical Reviewer
1. What Is a Large Language Model? Getting Started with Libraries and Environment Setup for Building an LLM from Scratch
Getting Started with the Foundations of Language ModelingWhat Is a Large Language Model (LLM)?The Attention Mechanism—Cornerstone of Modern Large Language Models (LLMs)What Are Attention Mechanisms?Detailed Explanation About How Self-Attention Works in TransformersWhy Attention Is Central to LLMsVariants and Innovations in Attention (As of May 2025)Challenges and Limitations of AttentionPractical Significance in LLMsThe Tools We Will Use to Build a Large Language Model from ScratchWhy Python for Building Large Language Models?Readable, Simple, and Accessible for All Skill LevelsExtensive Machine Learning Ecosystem and Strong Community SupportIntegrated Tools for Visualization, Experimentation, and Full ML PipelinesHigh Performance Through GPU Acceleration, Scalability, and Low-Level InteroperabilityProduction-Ready Deployment and Access to Pretrained Models for Transfer LearningWhy Jupyter Notebook for Building Large Language Models?Interactive, Iterative Coding with Immediate FeedbackUnified Environment for Code, Documentation, and VisualizationDeep Debugging Transparency and Real-Time InspectionOptimized for Learning, Collaboration, and Reproducible ResearchSeamless Integration with Scientific Libraries and Scalable EnvironmentsWhy PyTorch for Building Large Language Models?Dynamic, Flexible Execution for Fast Experimentation and DebuggingPythonic Design and Deep Integration with the Scientific EcosystemHigh-Performance GPU Acceleration and Parallel Computing at ScaleModular Architecture and a Rich, Research-Friendly EcosystemProduction-Ready Deployment, Interoperability, and Future-Proof DevelopmentWhy CUDA for Building Large Language Models?Massive Parallelism for Compute-Intensive OperationsSeamless Integration with Deep Learning FrameworksSupport for Mixed Precision TrainingOptimized Libraries for Deep LearningScalability for Distributed TrainingSupport for Advanced Hardware FeaturesFlexibility for Custom Kernels and OptimizationsRapid Prototyping and ExperimentationSupport for Inference OptimizationExtensive Ecosystem and Community SupportEnergy Efficiency and Cost SavingsFuture-Readiness and InnovationAccessibility for Diverse DevelopersSupport for Research and Custom ArchitecturesWhy NumPy and Matplotlib for Building Large Language Models?NumPy: Efficient Numerical Computations for Data ProcessingNumPy: Memory Efficiency for Large-Scale DataNumPy: Foundation for Preprocessing and Feature EngineeringNumPy: Support for Custom Algorithms and ResearchMatplotlib: Visualization for Model UnderstandingMatplotlib: Debugging and InterpretabilityMatplotlib: Customizable and Publication-Ready VisualsNumPy and Matplotlib: Seamless Integration with Python EcosystemNumPy and Matplotlib: Accessibility for BeginnersNumPy: Support for Dataset Creation and EvaluationMatplotlib: Support for Experiment TrackingNumPy and Matplotlib: Support for Research and InnovationNumPy and Matplotlib: Lightweight and Local-FirstNumPy and Matplotlib: Community and Ecosystem SupportTinyStories, Shakespeare, The Wizard of Oz, and OpenWebTextFour Key Datasets for Building and Understanding LLMsTinyStories: Learning LLM Fundamentals Through Simple NarrativesShakespeare: Stylized, Poetic, and Ideal for DemonstrationsThe Wizard of Oz: A Compact, Coherent Narrative for PrototypingOpenWebText: Large-Scale, Diverse, and Suitable for Real LLM TrainingHow These Datasets Work TogetherEthical and Practical ConsiderationsWhy PyLZMA for Building Large Language Models?Why PyLZMA Matters for LLM DevelopmentAdvanced ConceptsIn Summary
2. Foundational Concepts in LLM Development
Large Language Models Common ArchitectureTransformer Architecture: Conceptual and Mathematical Deep DiveInput EmbeddingPositional EncodingAttention MechanismScaled Dot-Product AttentionMultihead AttentionMasked Attention (Decoder)Feed-Forward Neural NetworkResidual Connections and Layer NormalizationOutput LayerLoss FunctionEncoder-Decoder InteractionComplexity AnalysisAdvantages of the TransformerLimitationsApplicationsVariants and ImprovementsThe Comprehensive Mathematics Behind Training Large Language ModelsExplaining the Purpose and ScopeDefining the Goal of Language ModelingNext-Token PredictionModeling Sequential ProbabilityDefining the Loss FunctionSoftmax for Probability DistributionPractical ConsiderationsMasked Language ModelingBidirectional Context ModelingComparison with Next-Token PredictionOptimization: Gradient-Based LearningGradient DescentIterative Parameter UpdatesStochastic Gradient DescentAdam OptimizerAdaptive OptimizationWhy Adam Works for LLMsBackpropagationComputing GradientsRole of Automatic DifferentiationRegularization TechniquesPreventing OverfittingDropoutRandomly Deactivating NeuronsWhy Dropout HelpsWeight DecayPenalizing Large WeightsImplementation in AdamLabel SmoothingSoftening Target DistributionsImpact on TrainingFine-Tuning and RLHFAdapting Pretrained ModelsSupervised Fine-Tuning (SFT)Task-Specific OptimizationDifferences from PretrainingReinforcement Learning from Human Feedback (RLHF)Aligning with Human PreferencesWhy RLHF Is EffectiveSummarizing the Mathematical FrameworkModern LLM Architecture Characteristics As of the End of 2024 and During 2025In Summary
3. Building a Tokenizer for the Transformers Architecture Model
What Is Tokenization?Why Is Tokenization Important?Types of TokenizersWord-Based TokenizersCharacter-Based TokenizersSubword-Based TokenizersRule-Based TokenizersByte-Pair Encoding (BPE): A Deep DiveOrigins of BPEBPE Algorithm Outline and Mathematical FormulationIdentify Frequent PairsReplace and RecordRepeat Until No GainsDecompression (Decoding)BPE Algorithm ExampleConcrete Example of the Encoding PartIteration 1Iteration 2Iteration 3Encoding New TextConcrete Example of the Decoding PartOptimization ObjectiveVocabulary Size and HyperparametersStep-by-Step Process of Building a BPE TokenizerInitialize the VocabularyCount Pair FrequenciesMerge the Most Frequent PairRepeat MergingTokenize New TextMap Tokens to IDsApplications of BPEAdvantages of BPELimitations of BPEComparison with Other Tokenization MethodsPractical ConsiderationsBPE Implementation WalkthroughTrain Method Code Breakdown—Step by StepStep 1: Preprocess the CorpusStep 2: Collect Base CharactersStep 3: Build Initial VocabularyStep 4: Map Text to Initial Token IDsStep 5: Learn MergesStep 6: Build GPT-2-Style Merge RanksThe Encode MethodThe Decode Method: One of the Most Important OnesStep 1: Reconstruct Raw String from TokensStep 2: Normalize Unicode and Remove Control CharactersStep 3: Clean Up Underscore MarkupStep 4: Clean Up Whitespace and PunctuationWhy This Method Is One of the Most ImportantSave and Load MethodsHelping FunctionsTesting Our Tokenizer for AccuracyTokenizer OutputAnalysis of the OutputTraining the TokenizerLearned MergesEncoding and Decoding ExamplesExample A: Simple SentenceExample B: Sentence with NewlineExample C: Special Token <mask>Example D: With BOS and EOSStep-by-Step BPE TraceSaving and Loading
4. RMS Normalization and Model Configuration
Model Parameters Configuration and Mathematical FoundationsMathematical NotationGlobal Tensor Shapes and Consistencynum_hidden_layers—The Depth of Abstractionvocab_size—The Universe of Symbolshidden_size—The Resolution of Thoughtintermediate_size—The Breathing Room of the Networkhead_dim—The Grain of Attentionnum_attention_heads—The Many Eyes of the Modelnum_key_value_heads—Sharing the Burdensliding_window—Constraining Attentioninitial_context_length—How Far the Model Can SeeThe Geometry of Position—rope_theta, rope_scaling_factor, rope_ntk_alpha, rope_ntk_betarope_theta—The Base Frequencyrope_scaling_factor—Stretching the Maprope_ntk_alpha—Balancing Long-Context Stabilityrope_ntk_beta—Controlling the Rate of Growthswiglu_limit—Taming the ActivationsWhat Is SwiGLU?Background on Related ConceptsHow SwiGLU WorksAdvantages and UsageSwiGLU Enhances FFN Expressiveness in GPT Architectures with Gated ActivationsRMS Normalization in GPT ArchitecturesWhat Is RMS Normalization?Why Is RMSNorm Used in GPT Architectures?How Is RMSNorm Used in GPT Architectures?Step-by-Step Implementation in a Transformer Block:RMS Normalization for Our Custom Large Language ModelComprehensive Explanation of RMSNorm CodeDetailed Code Explanation1. Import and Class Definition2. Initialization (__init__)3. Forward Pass (forward)Mathematical SummaryImplementation Details and Design ChoicesSummary—RMS Normalization and Model Configuration
5. Rotary Positional Embeddings: Integrating NTK and YaRN Scaling
Rotary Positional Embeddings: An In-Depth and Comprehensive ExplorationThe Fundamental Need for Positional Embeddings in TransformersHistorical Evolution of Positional EncodingsPretransformer Approaches: Sequential ProcessingEarly Transformer Positional EncodingsSinusoidal Positional EmbeddingsLearned Absolute Positional EmbeddingsRelative Positional EncodingsTransformer-XL Relative EmbeddingsT5 Relative BiasEmergence of Rotary Positional Embeddings (RoPE)Summary of Traditional Positional Embedding ApproachesAbsolute Positional EmbeddingsRelative Positional EmbeddingsAbsolute vs. Relative Positional EmbeddingsMathematical Formulation of RoPE2D Intuition and DerivationStarting in Two DimensionsThe Complex Number PerspectiveGeneralizing to High DimensionsAttention Decay and Frequency SelectionEfficient ImplementationAttention Matrix HeatmapConnection to Linguistic InterpretabilityAdvantages of RoPEApplications in Modern ModelsVariants and ExtensionsWhy Rotary Positional Embeddings Are Essential to Large Language ModelsSolving Permutation InvarianceSuperior Length ExtrapolationBalancing Absolute and Relative InformationEfficiency and ScalabilityNatural Attention DecayTraining BenefitsIndustry AdoptionRoPE Embeddings ImplementationFunction SignatureStep-by-Step Explanation1. Splitting the Embedding Tensor2. Preparing Cosine and Sine Tensors3. Applying RotationsOutputKey Features and Safety ConsiderationsExample WorkflowGeometric InterpretationRotaryEmbedding Class Definition and MethodsClass StructureDetailed Explanation1. Initialization (__init__)2. Computing Concentration and Inverse Frequencies (_compute_concentration_and_inv_freq)3. Computing Cosine and Sine Tensors (_compute_cos_sin)4. Forward Pass (forward)Key Features and Safety ConsiderationsExample WorkflowGeometric InterpretationWhat Is NTK (Neural Tangent Kernel) in the Context of RoPEImplementation in RotaryEmbeddingKey FeaturesWhat Is YaRN (Yet Another RoPE Extension)YaRN’s Scaling ApproachImplementation in RotaryEmbeddingKey FeaturesSummary

6. Scaled Dot-Product Attention Core—Sliding Window and Grouped Query Attention—The Core Behind All Transformer Models
What Is Scaled Dot-Product Attention (SDPA)?Historical Evolution and Contextual FoundationsIntuitive and Conceptual Underpinnings, Math FormulationsMasking MechanismsCausal MaskingPadding MaskingCustom Masks for Structured DataSparse Attention MasksLearned and Adaptive MasksBidirectional and Cross-Attention MasksSpecialized Attention FormsHierarchical AttentionAdaptive Attention SpanChallenges and Limitations of Masking MechanismsComplexity of Designing MasksThe Risk of Over-MaskingComputational Overhead of Generating and Applying MasksFuture Directions in MaskingThe Enduring Legacy of SDPACustom Implementation of SDPA—Sliding Window and Grouped Query Attention for Our LLMUnderstanding the Inputs and OutputsStep 1: Shape Inference, Broadcasting, and ReshapingStep 2: Computing Raw Attention ScoresStep 3: Constructing and Applying the Attention MaskStep 4: Incorporating Sink LogitsStep 5: Softmax Normalization and Output ComputationSummary
7. AttentionBlock with Rotary Embedding, GQA, Sliding Window, and Sink Tokens
Fundamentals of AttentionSelf-Attention in TransformersOperational PropertiesCausal Self-AttentionImplementation in TransformersRole in LLMsLimitations and ChallengesOptimizations and VariantsMultihead Attention in TransformersOperational PrinciplesIntegration in TransformersPositional ConsiderationsRole in LLMsLimitations and ChallengesOptimizations and VariantsAdvanced Considerations (up to 2025)What Is Grouped Query Attention, Sliding Window Attention, and Sink Tokens?Grouped Query Attention (GQA)Sliding Window Attention (SWA)Sink TokensIntegration in the Attention BlockIntegration of Attention Mechanism in Our Custom Large Language ModelExhaustive Explanation of the AttentionBlock PyTorch ModuleClass Definition and PyTorch IntegrationConstructor (__init__) MethodParameter Extraction and GQA ConfigurationValidation ChecksSliding Window and GQA GroupingNormalization LayerQKV Projection LayerOutput Projection LayerRotary Embedding InitializationSink Logits ParameterSoftmax Scaling FactorForward Pass (forward Method)Input Unpacking and Residual ConnectionPre-attention NormalizationQKV Projection and SlicingGQA ReshapingRoPE ApplicationCore Attention ComputationOutput Projection and Residual AdditionTraining and Inference BehaviorsPerformance OptimizationsInterpretabilityComparison to Other Attention MechanismsPractical Implementation NotesSummary
8. Multilayer Perceptron Block with Mixture of Experts (MoE) and SwiGLU
Mixture of Experts: A Comprehensive OverviewArchitecture of Mixture of ExpertsComponentsSparse ActivationVariants of MoEMathematical FoundationsTraining MechanismsLoss FunctionBackpropagation in MoELoad BalancingOptimization EnhancementsChallenges in TrainingApplications of Mixture of ExpertsAdvantages of MoELimitations of MoESwiGLU: An In-Depth ExplorationHistorical ContextThe Basics: From Activation Functions to Gated UnitsActivation FunctionsGated Linear Units (GLU)SwiGLU: Definition and StructureMathematical RepresentationParameter CountTraining MechanismsOptimization ConsiderationsChallengesIntegration in TransformersApplications of SwiGLUAdvantages of SwiGLULimitations of SwiGLUPractical ConsiderationsAdvanced Variants and ComparisonsVariants of SwiGLUComparisons with Other ActivationsMultilayer Perceptron Blocks with Mixture of Experts (MoE) and SwiGLU: A Comprehensive IntegrationIntegrated Architecture: MLP with MoE and SwiGLUArchitecture OverviewMathematical FoundationsTraining MechanismsForward and Backward PassLoad BalancingOptimization EnhancementsChallengesWhy They Work Together: RationaleScalability and EfficiencySpecialization and ExpressivenessGradient Flow and Training StabilityEmpirical PerformanceFlexibility Across DomainsAdvantagesHigh PerformanceComputational EfficiencyScalabilityRobust TrainingLimitationsTraining ComplexityHardware DependencyInterpretabilityMemory RequirementsPractical Implementation ConsiderationsArchitecture DesignFramework SupportTraining WorkflowHardware OptimizationEvaluationFuture DirectionsMultilayer Perceptron Block with Mixture of Experts (MoE) and SwiGLU for Our Large Language ModelIn-Depth Analysis of the MLPBlock with Mixture of Experts (MoE) and SwiGLUOverview of the MLPBlockInitialization ComponentsParameter Sharding and Memory ConsiderationsForward Pass: Step-by-Step ExecutionStep 1: Input and Residual ConnectionStep 2: RMS NormalizationStep 3: Gating Network and Top-k SelectionStep 4: Gathering Expert ParametersStep 5: First Projection (H → 2 * I_local)Step 6: SwiGLU ActivationStep 7: Second Projection (I_local → H)Step 8: Tensor ParallelismStep 9: Weighted Sum Across ExpertsStep 10: Residual Connection and OutputTraining DynamicsBackpropagationLoad BalancingOptimizationNumerical StabilityEdge Cases and RobustnessSummary
9. Transformer Block and Full Transformer Model—It’s Time to Put the Puzzle Together
The Role of the Transformer Block in Sequence ProcessingThe Architecture of the Transformers BlockQuery-Key-Value Projections and Multihead DecompositionRotary Position Embeddings: Geometric FoundationsScaled Dot-Product Attention: The Core MechanismThe Feed-Forward Block: Position-Wise TransformationsAlternative Activations: SwiGLU and Gated VariantsBlock Composition and Information FlowFull Model Architecture and Training DynamicsBuilding the Transformers Block for Our LLM from ScratchThe TransformerBlock ClassInitialization Method (__init__)Forward Pass MethodTransformer Class—The Complete Language ModelInitialization Method—Building the Full ArchitectureForward Pass—From Tokens to PredictionsStep 1: Embedding LookupStep 2: Processing Through Transformer LayersStep 3: Final NormalizationStep 4: Output Projectionfrom_checkpoint Class Method—Loading Pretrained ModelsStep-by-Step Loading ProcessImport Necessary ModulesDevice HandlingLoad Configuration from JSONInitialize Model ArchitectureLoad Model WeightsLoad State Dictionary into ModelError Handling and ValidationSet to Evaluation ModeArchitecture Design Choices and Modern PracticesAdvantages, Challenges, and Broader ImpactParallelism and ScalabilityTransfer Learning and Few-Shot GeneralizationInterpretability and Mechanistic UnderstandingFundamental ChallengesQuadratic Complexity and Context Length LimitationsData Efficiency and Sample ComplexityCatastrophic Forgetting and Continual LearningAlignment and ControlRobustness and Adversarial VulnerabilitiesFuture TrajectoriesArchitectural Innovations on the HorizonToward Artificial General IntelligenceSummaryAdvantages and Challenges
10. Dataset Preparation, Model Training, Token Generator for Inference and Prompting—The BIG Moment
Dataset Preparation for LLM TrainingThe Importance of Dataset QualityData Collection and SourcingData Cleaning and FilteringDeduplicationData Formatting and TokenizationDataset Composition and MixingBest Practices and RecommendationsPreparing Our DatasetText Characteristics and Classification of the DatasetGenre and Style IdentificationData Preparation Pipeline for This TextDataset Balancing ConsiderationsTraining Large Language ModelsModel ArchitectureTraining ObjectivesOptimizationDistributed TrainingInfrastructureTraining StagesChallengesHyperparameter SelectionCompute and EfficiencyAdvanced TechniquesPost-trainingEthical ConsiderationsSource Code for Training Our LLMUnderstanding the Code Step-by-StepPart 1: Environment Setup and ConfigurationPart 2: Dataset PreparationPart 3: Tokenizer Training or LoadingPart 4: Dataset Encoding and CachingPart 5: DataLoader SetupPart 6: Model Configuration and InitializationPart 7: Optimizer ConfigurationPart 8: Learning Rate SchedulingPart 9: Model Compilation (Optional)Part 10: The Training LoopPart 11: Saving the ModelUnderstanding Training DynamicsWhat the Model LearnsTokenGenerator for InferenceWhat Is TokenGenerator?What Is TokenGenerator?Core FunctionalityThe Generation ProcessSophisticated Anti-repetition System1. Windowed Repetition and Frequency Penalties2. No-Repeat N-gram Ban3. Self N-gram Ban (Bigram/Trigram Blocking)4. Dataset Anti-Copy Ban5. Self Anti-Copy Ban6. Variable-Period Loop DetectionAdvanced Sampling StrategyTypical SamplingTop-K SamplingTop-P (Nucleus) SamplingRobust FallbacksTemperature ControlWhy So Many Guardrails?Practical UsageKey Differences from TrainingThe Big Moment—Prompting Our ModelUnderstanding Inference Code and Model BehaviorSetup and InitializationBuilding Anti-copy IndicesText Cleaning UtilitiesLoop DetectionGeneration Function with Sophisticated ControlsInteractive InterfaceUnderstanding the Output—An Educational DemonstrationThe User QueryThe Generated Output—Three Revealing BehaviorsPhase 1: Accessing Learned KnowledgePhase 2: Pattern Repetition Under ConstraintsPhase 3: Exploring Low-Probability SpaceWhat This Demonstrates About Language Model DesignPrinciple 1: Model Scale and GeneralizationPrinciple 2: Training Data DiversityPrinciple 3: Training Duration and ComputePrinciple 4: Instruction Fine-Tuning and AlignmentThe Value of This DemonstrationWhat Pretraining Alone AchievesWhat Requires Additional Scale and TrainingHow Inference Mechanisms WorkPractical Lessons from This ExampleLesson 1: Match Model Scale to Task ComplexityLesson 2: Inference Constraints Have LimitsLesson 3: Training Data Determines CapabilitiesLesson 4: Hyperparameter Tuning MattersLesson 5: Multistage Training Is EssentialExploring Even Further—A Real Trained Model: TinyStories GPT-4 Version ImplementationTinyStories Dataset and ModelWhat Makes This Implementation DifferentLearning from the ImplementationKey Technical Differences from the Book’s CodeArchitecture and ScaleTokenizationDataset ManagementDistributed Training InfrastructureAdvanced Training FeaturesConfiguration ManagementGeneration and InferenceData Loading StrategyTraining Metrics and MonitoringOptimization DetailsCheckpoint ManagementWeb DeploymentWhy This Implementation Wasn’t Included in the BookThe Reality of Training CostsAccessibility and Learning GoalsThe Scaling GradientThe Value PropositionWhen to Make the InvestmentAppreciating What Was AchievedLooking Forward
11. Advanced Training and CUDA Kernels
The Journey from Raw Text to Intelligent Assistant—The Art and Science of LLM TrainingPretraining—Building the FoundationPretraining DataNext-Token PredictionArchitecture and Training SetupCompute Budget, Duration, and EvaluationMid-Training—Targeted Capability DevelopmentWhat Is Mid-Training?Why Mid-Training MattersLimitations of Pure PretrainingBenefits of Mid-TrainingTypes of Mid-TrainingDomain-Specific Mid-TrainingCapability-Specific Mid-TrainingData Quality EnhancementMid-Training MethodologyData CurationTraining ApproachPreventing Catastrophic ForgettingExamples of Successful Mid-TrainingMid-Training vs. Fine-TuningSupervised Fine-Tuning (SFT)—Teaching InstructionsThe Transition from Base to AssistantSFT Data: Instructions and DemonstrationsData FormatData SourcesDataset CompositionSFT Training ProcessObjective FunctionTraining HyperparametersData Quality in SFTThe Importance of Quality over QuantityQuality IndicatorsBalancing the SFT DatasetTask DistributionAvoiding OverfittingMultiturn Dialog TrainingThe Result: An Instruction-Following ModelReinforcement Learning from Human Feedback (RLHF)The Alignment ProblemThe Three Stages of RLHFReward Model TrainingPurposeData Collection ProcessTraining the Reward ModelReward Model OutputsReinforcement Learning OptimizationThe SetupThe Algorithm: Proximal Policy Optimization (PPO)Key Innovation: KL PenaltyChallenges in RLHFReward HackingReward Model LimitationsTraining InstabilityPractical ConsiderationsComputational CostHuman LabelingResults of RLHFAlternative Alignment ApproachesDirect Preference Optimization (DPO)The InnovationHow DPO WorksAdvantages of DPOLimitationsConstitutional AI (CAI)PhilosophyTwo-Stage ProcessConstitutional Principles ExampleAdvantagesReinforcement Learning from AI Feedback (RLAIF)Core IdeaWhen RLAIF Works WellLimitationsIterative ApproachesIterative RLHFOnline LearningHybrid ApproachesPost-Training Techniques and RefinementsContext DistillationSelf-Improvement TechniquesSelf-CritiqueIterative RefinementRed Teaming and Adversarial TrainingRed TeamingAdversarial TrainingCapability-Specific Fine-TuningMultiobjective OptimizationEvaluation and BenchmarkingEvaluation During PretrainingIntrinsic MetricsDownstream TasksEvaluation During Supervised Fine-TuningInstruction FollowingTask PerformanceStyle and FormatEvaluation During Alignment TrainingPreference ModelingSafety EvaluationsHuman EvaluationComprehensive BenchmarksKnowledge and ReasoningCodingSafety and AlignmentMultilingualThe Limitations of BenchmarksBenchmark SaturationGap Between Benchmarks and Real-World UseSolutionsPractical Considerations and Best PracticesData Is KingComputational Resource ManagementCost-Benefit AnalysisStrategic DecisionsPreventing DegradationCommon PitfallsPrevention StrategiesScaling Laws and EfficiencyCompute-Optimal TrainingEfficiency TechniquesResponsible AI ConsiderationsThroughout the Training PipelineRed LinesThe Future of LLM TrainingEmerging TrendsMultimodal TrainingLonger Context WindowsContinuous LearningMore Efficient Training MethodsFew-Shot and Zero-Shot AlignmentSelf-Supervised AlignmentMixture of ExpertsBetter EvaluationMore Robust BenchmarksAutomatic EvaluationDemocratizationSmaller, More Efficient ModelsOpen-Source ProgressBetter Tools and InfrastructureTheoretical UnderstandingWhy Does It Work?ControllabilityTraining Neural Networks with CUDA Kernels and Modern FrameworksUnderstanding CUDA and GPU ComputingWhat Is CUDA?Why GPUs for Deep Learning?The CPU vs. GPU ParadigmCUDA Kernels ExplainedWhat Is a CUDA Kernel?Thread HierarchyKernel SyntaxMemory HierarchyCUDA Kernels in Neural Network TrainingWhere Kernels Are UsedThe Training Loop at Kernel LevelWhy Custom Kernels MatterPractical CUDA Kernel ExamplesExample 1: Element-Wise ReLU ActivationExample 2: Matrix Multiplication (Naive Implementation)Example 3: Optimized Matrix Multiplication with Shared MemoryExample 4: Softmax with Numerical StabilityExample 5: Custom Fused Kernel—LayerNorm + GELUIntegration with Deep Learning FrameworksPyTorch Custom CUDA ExtensionsTriton: High-Level GPU ProgrammingPerformance Optimization StrategiesMemory CoalescingOccupancy OptimizationKernel FusionAsynchronous OperationsUsing Tensor CoresChapter 7: Real-World Training PerformanceProfiling and Bottleneck IdentificationFlashAttention ExampleTraining at ScalePractical Tips for Working with CUDA KernelsWhen to Write Custom KernelsDevelopment WorkflowDebugging CUDA KernelsCommon PitfallsAppendix: Glossary of Terms
Index

Content preview from Building Large Language Models from Scratch: Design, Train, and Deploy LLMs with PyTorch

D. GrigorovBuilding Large Language Models from Scratchhttps://doi.org/10.1007/979-8-8688-2297-1_7

7. AttentionBlock with Rotary Embedding, GQA, Sliding Window, and Sink Tokens

Dilyan Grigorov¹

Dilyan Grigorov

Email: dido.grigorov@gmail.com

Attention mechanisms are the cornerstone of modern large language models (LLMs), underpinning their ability to process and generate human-like text. Introduced in the 2017 paper “Attention is All You Need” by Vaswani et al., the transformer architecture replaced recurrent neural networks (RNNs) with attention-based mechanisms, enabling parallel computation and direct modeling of relationships between tokens.

Unlike RNNs, which struggled with vanishing gradients ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Build a Large Language Model (From Scratch), Video Edition

Publisher Resources

ISBN: 9798868822971

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Building Large Language Models from Scratch: Design, Train, and Deploy LLMs with PyTorch

by Dilyan Grigorov

7. AttentionBlock with Rotary Embedding, GQA, Sliding Window, and Sink Tokens

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.