book

Build a Reasoning Model (From Scratch)

Name: Build a Reasoning Model (From Scratch)
Author: Sebastian Raschka
ISBN: 9781633434677

by Sebastian Raschka

June 2026

Intermediate

440 pages

11h 28m

English

Manning Publications

Read now

Unlock full access

Build a Reasoning Model (From Scratch)
copyright
contents
dedication
preface
acknowledgments
about this book
about the author
about the cover illustration
1 Understanding reasoning models
1.1 Defining reasoning in the context of LLMs1.2 Understanding the standard LLM training pipeline1.3 Improving LLM reasoning with training and inference techniques1.4 Pattern matching vs. logical reasoning1.5 Simulating reasoning without explicit rules1.6 Why build reasoning models from scratch?1.7 A road map to building reasoning models from scratch

2 Generating text with a pretrained LLM
2.1 Introducing LLMs for text generation2.2 Setting up the coding environment2.3 Understanding hardware needs and recommendations2.4 Preparing input texts for LLMs2.5 Loading pretrained models2.6 Understanding the sequential LLM text generation process2.7 Coding a minimal text generation function2.8 Faster inference via KV caching2.9 Faster inference via PyTorch model compilation
3 Evaluating reasoning models
3.1 Building a math verifier3.2 Loading a pretrained model to generate text3.3 Implementing a wrapper for easier text generation3.4 Extracting the final answer box3.5 Normalizing the extracted answer3.6 Verifying mathematical equivalence3.7 Grading answers3.8 Loading the evaluation dataset3.9 Evaluating the model
4 Improving reasoning with inference-time scaling
4.1 Introduction to inference-time scaling4.2 Loading a pretrained model4.3 Generating better responses with chain-of-thought prompting4.4 Controlling output diversity with temperature scaling4.4.1 Understanding the process of selecting the next token4.4.2 Rescaling token scores (logits) via a temperature parameter4.4.3 Sampling the next token from a probability distribution4.4.4 Adding temperature scaling to the text generation function4.5 Balancing diversity and coherence with top-p sampling4.5.1 Selecting a subset of top-p tokens4.5.2 Adding a top-p filter to the text generation function4.6 Improving response accuracy with self-consistency
5 Inference-time scaling via self-refinement
5.1 Scoring and iteratively improving model responses5.2 Loading a pretrained model5.3 Scoring LLM responses with a rule-based score5.4 Understanding token probability scores5.5 From token probability scores to log probabilities5.6 Scoring model confidence with log probabilities5.7 Self-refinement through iterative feedback5.8 Coding the self-refinement loop
6 Training reasoning models with reinforcement learning
6.1 Introduction to RL for LLMs6.1.1 The original RL pipeline with human feedback (RLHF)6.1.2 From human feedback to verifiable rewards6.2 RLVR using GRPO6.2.1 High-level GRPO intuition via a chef analogy6.2.2 The high-level GRPO procedure6.3 Loading a pretrained model6.4 Loading a MATH training subset6.5 Sampling rollouts6.6 Calculating rewards6.7 Preparing learning signals from rollouts via advantages6.8 Scoring rollouts with sequence log probabilities6.9 From advantages to policy updates via the GRPO loss6.10 Putting everything together in a single GRPO function6.11 Implementing the GRPO training loop6.12 Loading and evaluating saved model checkpoints
7 Improving GRPO for reinforcement learning
7.1 Improving GRPO7.2 Tracking GRPO performance metrics7.2.1 Executing a GRPO training run7.2.2 Inspecting the GRPO training run7.3 Tracking more advanced GRPO performance metrics7.3.1 Advantage tracking7.3.2 Entropy tracking7.3.3 Plotting additional GRPO metrics7.4 Stabilizing sequence-level GRPO using clipped policy ratios7.4.1 Computing clipped policy ratios7.4.2 Training with clipped policy ratios7.5 Controlling how much the model changes with a KL term7.5.1 Implementing the KL loss term7.5.2 Training with a KL loss term7.6 Adding an explicit format reward7.6.1 Using <think> tokens7.6.2 Training a model to emit <think> tokens7.6.3 More GRPO modifications, tips, and tricks
8 Distilling reasoning models for efficient reasoning
8.1 Introducing model distillation for reasoning tasks8.2 Generating a dataset for reasoning distillation8.3 Loading the MATH training dataset for distillation8.4 Building training examples8.4.1 Loading and understanding the tokenizer8.4.2 Formatting and tokenizing the dataset8.4.3 Filtering and splitting the dataset8.5 Loading a pretrained model8.6 Computing the training and validation losses8.7 Implementing the training loop for distillation8.8 Evaluating the distilled model8.9 Future directions for reasoning models8.10 Conclusions8.10.1 What’s next8.10.2 Staying up to date in a fast-moving field
appendix A References and further reading
A.1 Chapter 1: Understanding reasoning modelsA.1.1 ReferencesA.1.2 Further readingA.2 Chapter 2: Generating text with a pretrained LLMA.2.1 ReferencesA.2.2 A.2.2 Further readingA.3 Chapter 3: Evaluating reasoning modelsA.3.1 ReferencesA.3.2 Further readingA.4 Chapter 4: Improving reasoning with inference-time scalingA.4.1 ReferencesA.4.2 Further readingA.5 Chapter 5: Inference-time scaling via self-refinementA.5.1 ReferencesA.5.2 Further readingA.6 Chapter 6: Training reasoning models with reinforcement learningA.6.1 ReferencesA.6.2 Further readingA.7 Chapter 7: Improving GRPO for reinforcement learningA.7.1 ReferencesA.7.2 Further readingA.8 Chapter 8: Distilling reasoning models for efficient reasoningA.8.1 ReferencesA.8.2 Further readingA.9 Appendix F: Common approaches to LLM evaluationA.9.1 ReferencesA.9.2 Further reading
appendix B Exercise solutions
B.1 Chapter 2B.2 Chapter 3B.3 Chapter 4B.4 Chapter 5B.5 Chapter 6B.6 Chapter 7B.7 Chapter 8
appendix C Qwen3 LLM source code
C.1 RMSNormC.2 Feedforward moduleC.3 Rotary position embeddingsC.4 Grouped query attentionC.5 Transformer blockC.6 Main model codeC.7 KV cacheC.8 TokenizerC.9 Using the model
appendix D Using larger LLMs
D.1 Larger dense Qwen3 configurationsD.2 Downloading larger checkpoints overviewD.3 Loading a larger base modelD.4 Loading a larger reasoning variantD.5 Practical recommendations
appendix E Batching and throughput-oriented execution
E.1 Why batching helpsE.2 Running batched generationE.3 Padding and attention masksE.4 Chapter 3: Batched MATH-500 evaluationE.5 Chapter 4: Batched self-consistency samplingE.6 Chapter 6: Batched GRPO rolloutsE.7 Chapter 8: Batched distillationE.8 Single-sequence vs. batch generation
appendix F Common approaches to model evaluation
F.1 Understanding the main evaluation methods for LLMsF.2 Evaluating answer-choice accuracyF.2.1 Loading the modelF.2.2 Checking the generated answer letterF.3 Using verifiers to check answersF.4 Comparing models using preferences and leaderboardsF.5 Judging responses with other LLMsF.5.1 Implementing the LLM-as-a-judge approach in OllamaF.5.2 Evaluating responses with a grading rubric
appendix G Building a chat interface
G.1 Installing ChainlitG.2 Running the code as a scriptG.3 Downloading the scriptsG.4 The regular single-turn scriptG.5 Running the single-turn scriptG.6 The multi-turn interfaceG.6.1 What multi-turn meansG.6.2 The multi-turn variantG.6.3 How the multi-turn script uses historyG.6.4 Recommendations

Content preview from Build a Reasoning Model (From Scratch)

7 Improving GRPO for reinforcement learning

This chapter covers

Interpreting training curves and evaluation metrics
Preventing the model from exploiting the reward signal
Extending task-correctness rewards with additional response-formatting rewards

Previously, we implemented the GRPO algorithm for reinforcement learning with verifiable rewards (RLVR) from end to end. Now, as shown in figure 7.1, we’ll pick up from that baseline and focus on what happens when we run longer training.

Figure 7.1 A model of the topics covered in this book. This chapter provides deeper coverage of the GRPO algorithm for reinforcement learning with verifiable ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Agentic Architectural Patterns for Building Multi-Agent Systems

Publisher Resources

ISBN: 9781633434677Publisher Support Other Publisher Website Purchase Link

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Build a Reasoning Model (From Scratch)

by Sebastian Raschka