book

Reinforcement Learning from Human Feedback

Name: Reinforcement Learning from Human Feedback
Author: Nathan Lambert
ISBN: 9781633434301

by Nathan Lambert

August 2026

Intermediate

312 pages

9h 21m

English

Manning Publications

Read now

Unlock full access

Reinforcement Learning from Human Feedback
copyright
contents
foreword
preface
acknowledgments
about this book
about the author
about the cover illustration
Part 1. Overview

1 Introduction
1.1 RLHF in three steps1.2 What does RLHF do?1.3 Walkthrough of an RLHF recipe1.4 An intuition for post-training1.5 How we got here1.6 Future of RLHF
2 A tiny history of RLHF
2.1 Origins to 2018: RL on preferences2.2 2019 to 2022: RL from human preferences on language models2.3 2023 to the present: The ChatGPT era
3 Training overview
3.1 Problem formulation3.1.1 A simple example: The thermostat3.1.2 Classic RL example: CartPole3.1.3 Manipulating the standard RL setup3.1.4 Fine-tuning and regularization3.1.5 Optimization tools3.1.6 Subtle advantages of RL in post-training language models3.2 Canonical training recipes3.2.1 InstructGPT3.2.2 Tülu 33.2.3 DeepSeek R1
Part 2. Core training methods
4 Instruction fine-tuning
4.1 Chat templates and the structure of instructions4.2 Best practices for instruction tuning4.3 Implementation details
5 Reward modeling
5.1 Training a Bradley–Terry reward model5.2 Implementation example5.3 Reward model variants5.3.1 Preference margin loss5.3.2 Balancing multiple comparisons per prompt5.3.3 K-wise loss function5.4 Outcome reward models5.5 Process reward models5.6 Comparing reward model types (and value functions)5.6.1 Inference across reward model types5.7 Generative reward modeling (a.k.a. LLM-as-a-judge)5.8 Further reading
6 Reinforcement learning
6.1 The role of reinforcement learning in RLHF6.2 Policy gradient algorithms6.2.1 Deriving the policy gradient6.2.2 Vanilla policy gradient6.2.3 REINFORCE6.2.4 REINFORCE leave one out6.2.5 Proximal policy optimization6.2.6 Understanding the PPO objective6.2.7 Value functions and PPO6.2.8 Group relative policy optimization6.2.9 Group sequence policy optimization6.2.10 Clipped importance sampling policy optimization6.2.11 Comparing algorithms6.3 Implementation6.3.1 Policy-gradient basics6.3.2 Loss aggregation tradeoffs6.3.3 Asynchronous RL systems6.3.4 Truncated importance sampling6.3.5 Example: PPO6.3.6 Example: GRPO6.3.7 RLOO vs. GRPO6.4 Auxiliary topics6.4.1 Generalized advantage estimation6.4.2 Double regularization6.5 Further reading
7 Reasoning and inference-time scaling
7.1 The role of RLVR7.2 The origins of new reasoning models7.2.1 Why does RL work now?7.2.2 RL training vs. inference-time scaling7.2.3 The future (beyond reasoning) of RLVR7.3 Understanding reasoning training methods7.3.1 Reasoning research before OpenAI o1 or DeepSeek R17.3.2 Early reasoning models7.3.3 Common practices in training reasoning models7.4 Looking ahead
8 Direct-alignment algorithms
8.1 Direct preference optimization8.1.1 How DPO works8.1.2 DPO derivation8.2 Numerical concerns, weaknesses, and alternatives8.3 Implementation details8.4 DAAs with synthetic preference data8.5 DAAs vs. RL: Online vs. offline data
9 Rejection sampling
9.1 Training process, step by step9.1.1 Generating completions9.1.2 Scoring completions9.1.3 Fine-tuning9.2 Implementation details9.3 Related: Best-of-N sampling
Part 3. Data and preferences
10 The nature of preferences
10.1 When preference replaces correctness10.2 The origins of RLHF and preferences10.3 Specifying objectives: From logic of utility to reward functions10.4 Tools for optimizing utility10.5 Complexity of optimizing preferences
11 Preference data
11.1 Why we need preference data11.2 Collecting preference data11.2.1 Interfaces11.2.2 Rankings vs. ratings11.2.3 Multiturn data11.2.4 Structured preference data11.2.5 Sourcing and contracts11.3 Bias: Things to watch out for in data collection11.4 Open questions in RLHF preference data
12 Synthetic data
12.1 The roles of synthetic data12.2 Distillation with synthetic data12.3 The path to on-policy, teacher–student distillation12.3.1 Adapting knowledge-distillation for LMs12.3.2 From offline to on-policy distillation12.3.3 Modern OPD variants12.4 AI feedback12.4.1 Balancing AI and human feedback data12.4.2 Building specific LLMs for judgment12.5 Constitutional AI12.5.1 Further reading on CAI12.6 Rubrics: Prompt-specific AI feedback for training
Part 4. Applications and advanced topics
13 Tool use and function calling
13.1 Tool-use overview13.2 Interweaving tool calls in generation13.3 Multistep tool reasoning13.4 Model Context Protocol13.5 Implementation details
14 Over-optimization
14.1 Qualitative over-optimization14.1.1 Managing proxy objectives14.1.2 Over-refusal and “too much RLHF”14.2 Quantitative over-optimization14.3 Misalignment and the role of RLHF
15 Regularization
15.1 KL divergence in RL optimization15.1.1 Reference model to generations15.1.2 Implementation example15.2 Implicit regularization15.2.1 SFT memorizes, RL generalizes15.2.2 Retaining by doing: On-policy data mitigates forgetting15.2.3 RL’s razor: Why online RL forgets less15.3 Other types of regularization15.3.1 Pretraining gradients15.3.2 Margin-based regularization
16 Evaluation
16.1 Prompting formatting16.1.1 Few-shot prompting and log-likelihood scoring16.1.2 Chain-of-thought prompting16.1.3 Zero-shot instruction following16.1.4 Reasoning-era evaluation prompts16.2 Why many external evaluation comparisons are unreliable16.3 How labs actually use evaluations internally to improve models16.4 Contamination16.5 Tooling
17 Crafting model character and products
17.1 Character training17.1.1 Persona vectors17.1.2 The assistant axis17.1.3 Persona subnetworks17.2 Model specifications17.3 Product cycles and what’s next for RLHF
A. Definitions
A.1 Language modeling overviewA.2 Machine learningA.3 Natural language processingA.4 Reinforcement learningA.5 RLHF-onlyA.6 Extended glossary
B. Beyond “just style”
C. Practical issues
C.1 Compute costs of post-trainingC.2 Evaluation varianceC.3 Managing training performance varianceC.4 Identifying bad training jobs
references

Content preview from Reinforcement Learning from Human Feedback

14 Over-optimization

This chapter covers

Understanding and recognizing over-optimization
Qualitative failures: over-refusal, sycophancy, and repetitiveness
The relationship between KL distance and model quality

A core lesson we learn when using reinforcement learning heavily in our domain is that it is a very strong optimizer, which causes it to pull all the possible increase in reward out of the environment. In modern ML systems, especially with language models, we’re using somewhat contrived notions of environment: the models generate completions (the actions), and an external verifier (that is, a reward model or a scoring function) provides feedback. In this domain, it is common for over-optimization to occur: the RL optimizers ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781633434301Publisher Support Other Publisher Website Purchase Link

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Reinforcement Learning from Human Feedback

by Nathan Lambert