The Two Pillars of Post-training: Reinforcement Learning and Supervised Fine-Tuning

This is the second article in Sharon Zhou’s post-training series. Read part 1 here.

In the first post of this series, you learned how post-training closed the fundamental gap in usability of LLMs by making them behave in a certain way. In this post, you’ll explore specific techniques you can use to change a model’s behavior: namely, reinforcement learning (RL) and supervised fine-tuning (SFT).

Reinforcement learning teaches the model by letting it try things and telling it which attempts were better or worse—the model learns by experimentation and feedback. Supervised fine-tuning teaches the model by showing it examples of good behavior—the model learns by imitations. Both have deep roots in AI and machine learning literature historically, but their application to LLMs, and particularly to making LLMs behave well, is what makes modern post-training work. Nearly everything that happens in post-training is the result of some combination of these two approaches.

Reinforcement learning (RL): Learning from feedback

The overall gist of reinforcement learning goes like this:

1. The model gets a prompt.
2. The model generates a response.
3. The model’s response is graded. The grade is called a reward. A positive reward is good, and a negative reward is bad.
4. The model’s weights are updated to make high-reward responses more likely and low-reward responses less likely.

One of the most important questions is: Where does the reward come from?

Verifiers

The easiest way to get a reward is a function that can output a reward, for example a checker for whether the generated code compiles or whether the generated math problem was solved correctly. This automated check is a verifier. The ideal verifiers are fast, cheap, and perfectly reliable within their domain. Think coding challenges, math problems, or factual questions. For tasks with objectively correct answers, you can just write a function that checks the output.

The limitation is probably obvious: Verifiers only work when you can define “correct” programmatically or hit an API to return the right results. That covers a lot of useful territory, but it doesn’t help you train a model to be helpful, nuanced, or pleasant to talk to.

There are subtler limitations too. Not all verifiers are fast. Your model might propose a novel drug combination, but verifying its validity could take years of lab work. Generated GPU code might need hours or days of performance benchmarking. When verification is expensive, you face a trade-off: Use the slow-but-accurate verifier sparingly, or substitute a faster proxy that’s slightly less reliable but keeps training moving.

Human feedback, RLHF, and reward models

Humans can offer strong reward signals that, in aggregate, align with human preferences that might be more subtle and hard to encode programmatically. However, it’s prohibitively inefficient to have humans in the loop for every training datapoint, especially as the model is continuously updating its weights after it receives rewards as feedback, so the model’s responses would change over time. You can’t really prepare the data ahead of time. So instead, the InstructGPT paper, which informed ChatGPT’s development, implements reinforcement learning from human feedback (RLHF) by training a separate model to mimic human feedback. This model is called a “reward model.”

The input of the reward model is a prompt and model response and its output is a scalar reward (positive or negative) that mimics how a person would rate that response. You can train a reward model in multiple ways. The simplest is to have people grade the model outputs with a score, for example 1–5 stars or a number out of 100%. However, people are rarely consistent at these types of tasks: One person’s 2 is another’s 5, and even the same person drifts over time.

Another simple way is to offer two model responses in comparison and ask, “Which one is better?” This is a much easier, more reliable judgment for people to make. Interannotator agreement is significantly higher for comparisons than for absolute ratings.

Training a model using pairwise comparisons is also simple. You can then use cross-entropy loss over pairs, which pushes the reward of the preferred response higher than the unpreferred one. This works great because it means the reward model can learn from signals like “A is better than B” but can learn to output absolute scores for the reward.

To make the process of collecting pairwise comparisons from people more efficient, the InstructGPT’s implementation of RLHF included showing labelers 4–9 different model outputs from a single prompt and asking them to rank those preferences. This would effectively result in 6–36 pairwise comparisons for a given ranking. Not bad; that’s efficient data labeling! They used ~33K prompts, so that would roughly translate to anywhere from 200K to 1.2M comparisons to train the reward model.

After training, the reward model would be an automated judge during RL training, providing scalar rewards for responses. The language model then optimizes against this reward model’s scores. This means the better the reward model, the more aligned the resulting model would be.

LLM as judge

So you need a reward: Why not use an LLM? LLM-as-judge, sometimes called RLAIF (reinforcement learning from AI feedback), scales much better than human annotation while still being able to evaluate subjective qualities like helpfulness, clarity, and tone. But it inherits whatever biases or blind spots the judge model has, and can be more easily gamed. If the judge tends to prefer verbose answers, the trained model will learn to be verbose.

One effective approach is to break the judgment into multiple LLM calls, each focused on a different aspect of the response, like a rubric. Instead of asking one LLM call “How good is this response?” you might have separate calls evaluating factual accuracy, clarity of explanation, appropriate tone, and completeness. Each dimension gets its own score, and you combine them into a final reward. This is more robust than a single holistic judgment because it’s harder for the model to game all dimensions at once, and it gives you fine-grained control over what you’re optimizing for. You can weigh the dimensions differently depending on what matters most for your use case, and adjust those weights over time as your priorities shift. For example, accuracy is worth 3x as much as tone.

Combining human feedback with LLM-as-judge, Anthropic’s Constitutional AI (CAI) is a method for training reward models from AI-generated comparisons, based on a human-written set of principles. What this means is that you can give an LLM a set of principles, which Anthropic calls a “constitution,” and have it critique and revise its own outputs based on those principles. For example, a principle might say “choose the response that is least likely to be harmful” or “prefer the answer that is most helpful while being honest.” The model generates pairs of responses, uses the constitution to decide which is better, and those AI preferences are used to train the reward model. This means you can encode your values explicitly as written principles in the Constitution rather than implicitly through thousands of human annotations, making it easier to audit, agree on, and update what the model is being trained to do.

RL algorithms

Once you have a reward, it’s time to update the model’s weights. But you can’t just predict the next token, because there isn’t one. All you have is a value for the response the model gave. This is where RL algorithms come in. These algorithms are ways to take the reward and turn it into a meaningful, and ideally stable, training signal for the model to learn. There are several, and the field is moving fast, but a few fundamental ones are worth understanding.

REINFORCE

REINFORCE is the simplest starting point. The idea is to generate a response, score it, and if the reward was high, nudge the model to make that response more likely. If the reward was low, nudge it to make that response less likely. It’s conceptually easy to grok but noisy and difficult in practice because it turns out that the signal from a single response can point the optimization in unhelpful directions, and the variance in the gradients makes training slow and unstable. PPO was designed to fix these exact problems.

PPO (proximal policy optimization)

PPO is what OpenAI used in the original ChatGPT work and was for a while the default algorithm for RLHF. In RL terminology, the model is the “policy,” or the thing that takes actions by outputting tokens, in an environment which is simply the conversation context.

PPO improves on REINFORCE by being more careful about how big each update is. Rather than taking whatever gradient the reward suggests, PPO clips the update so the model can’t change too much in a single step. This makes training significantly more stable. The clipping keeps updates “proximal.”

PPO is also an online algorithm, meaning the model generates fresh responses during training, gets them graded, and updates from that feedback in a continuous loop. So the model keeps learning from its own current behavior rather than from a static dataset. It can explore and improve in ways that offline methods (that only collect data once beforehand) can’t.

Notably, PPO uses a “critic,” or a separate model that predicts the expected total reward from any point during generation, and is trained with the policy. This helps reduce noise in training, because it gives you a baseline: Instead of just knowing “this response got a reward of 7” and having no idea if that’s good or bad, the critic might predict “you’d normally get a 5 here,” so the actual training signal (called an “advantage”) becomes “+2, better than expected.” This dramatically reduces noise compared to REINFORCE.

The downside is complexity. Now you’re training two models (the main model and the critic), and the whole pipeline involves generating responses, grading them with a reward model, estimating how good the grades are relative to the critic, and updating both models. It works, but it’s a lot of moving parts. This makes it harder to tune or debug when something goes wrong, and harder to set up the infrastructure.

DPO (direct preference optimization)

DPO takes a different approach that avoids RL entirely but optimizes the same underlying objective as the standard RLHF formulation. Researchers found that there’s a mathematical relationship between the optimal reward model and the optimal main model (policy), and you can collapse the two-step process into one. This means that you can take the same pairwise comparison data (“model response A is better than model response B”) and use it to update the main model directly, without a reward model. Yes, this means good old supervised learning on that pairwise data.

In theory, under ideal conditions, DPO and PPO-based RLHF converge to the same global optimum. Those ideal conditions include a perfect reward model, infinite preference data covering the full output distribution, and the reference policy matching the data-generating distribution. However, these rarely hold in practice, and several empirical studies have shown meaningful performance gaps between DPO and online RL methods on harder tasks, partly because DPO can’t explore beyond its fixed dataset. That said, it’s still a very promising technique.

The simplicity of DPO is attractive: supervised fine-tuning on pairwise data with no reward model to train and no RL loop to stabilize. As a result, DPO has become very popular, especially among smaller teams, because it’s much easier to implement and debug. However, the trade-off is that DPO is less flexible, because it works directly from a fixed dataset of preferences. This means it can’t explore and discover novel behaviors the way online RL methods can. It only learns from the comparisons you already have.

Newer online variants of DPO have addressed this by generating fresh responses during training, but at that point you’re reintroducing some of the infrastructure complexity that made DPO appealing to avoid in the first place.

GRPO (group relative policy optimization)

Introduced by DeepSeek, GRPO takes another stab at simplifying PPO. Instead of needing a separate critic model, GRPO generates a group of responses to the same prompt and uses the relative rewards within that group to figure out which responses were better or worse—basically normalizing within that group. If you generate eight responses and three of them score well, those three get reinforced and the others get pushed down, and the baseline (which the critic was in charge of previously) is just the group average. This eliminates the critic entirely while still getting a useful training signal. It’s simpler than PPO but still online (the model generates fresh responses during training), so it can explore in ways DPO can’t. GRPO got a lot of attention because of its role in training DeepSeek’s reasoning models.

There are many more algorithms and variants, and new ones appear regularly. The field hasn’t converged on a method (and likely won’t for some time), and different algorithms suit different situations. DPO is great when you have good preference data and want simplicity. PPO remains strong when you need online exploration and have the engineering resources to manage the complexity. GRPO offers an appealing middle ground. In practice, teams often try multiple approaches and pick what works best for their specific use case and reward signal.

RL post-training is also less stable than supervised learning, which we’ll cover next. The loss curves are noisier, the hyperparameters are more sensitive, and the training can diverge if not carefully managed. Practitioners typically constrain the RL updates with a penalty that prevents the model from drifting too far from its starting point. The most common approach is a KL divergence penalty that keeps the fine-tuned model’s output distribution close to the base (or SFT) model’s distribution. This acts as a regularizer: It lets the model improve its behavior while preventing it from forgetting what it learned in pretraining or collapsing into degenerate patterns.

Supervised fine-tuning (SFT): Teaching by demonstration

Supervised fine-tuning is more straightforward. You show the model examples of ideal responses, and train it to reproduce them. In practice, this means collecting a dataset of {prompt, ideal response} pairs and continuing to train the model’s weights using the same next-token prediction objective from pretraining, but now on this curated dataset instead of the broad pretraining dataset. The one difference is that the loss is computed only on the response tokens, not the prompt tokens, so the model learns to generate good responses given prompts, not to generate prompts.

The simplicity is the point. There’s no reward model to train, no critic to stabilize, and no policy gradient variance to worry about. However, it’s also limited by the data you can collect. That can get expensive and difficult to scale.

The quality of your SFT model is directly determined by the quality of your demonstrations. The model is learning to copy what you show it, so every quality issue in the data becomes a quality issue in the final model.

Human demonstrations

The most direct approach is to hire skilled people to write high-quality responses to a diverse set of prompts. This is the gold standard. You can easily control your dataset here, and you can get exactly what you want, written to your specifications. The original InstructGPT paper from OpenAI contracted 40 labelers, writing demonstrations and ranking outputs.

The disadvantage is, probably obviously, cost and scale. Good demonstrations are expensive, especially tasks requiring domain expertise like having doctors write an ideal prescription for a patient or a rocket scientist telling you how to put satellites on Mars. And even expert annotators are inconsistent. They have bad days, they get tired, and they interpret instructions differently from each other. At scale, this inconsistency can accumulate, though labeling companies manage and sell processes to make crowdwork more effective at scale.

Synthetic data

Synthetic data scales far better than human annotation. You can generate millions of demonstrations cheaply and quickly. The Stanford Alpaca project famously fine-tuned Llama on only 52,000 demonstrations generated by text-davinci-003 (part of the GPT-3.5 model family, though not ChatGPT) and was able to get qualitatively similar behavior to text-davinci-003 with a much smaller budget (though it was on a narrow evaluation of only ~250 examples—still an exciting result for small open models for research).

Many open source models have used variants of this approach. However, there’s also a practical consideration around terms of service. Some model providers restrict using their outputs to train competing models, and this has become an increasingly heated area of debate as models compete at the frontier. Know the rules before you build your pipeline.

Curated data with synthetic transformations

Sometimes the best demonstrations already exist. Customer support logs, internal documentation, expert Q&A forums, edited writing samples. If you have access to high-quality human-generated content that matches the behavior you want or is close to it, you can use LLMs to transform that data into prompt-response pairs. This has the advantage of being grounded in real use cases rather than fully synthetic scenarios.

The work is in the curation, and sometimes it might be easier to generate from scratch based on a few few-shot examples. Raw data is messy: Support logs contain errors, forums contain misinformation, and real conversations meander. You need to filter, clean, and reformat aggressively, but you can build an LLM pipeline to do those steps. If you have a good source and invest in the LLM curation pipeline, this can be extremely effective, especially for domain-specific applications.

Rejection sampling

Sometimes the best training signal is already inside the model and you just need to find it. Rejection sampling works by generating many possible responses to a prompt, scoring them with some quality metric, and keeping only the top performers. The quality metric can be a reward model, a rule-based check, or even a stronger model acting as a judge.

Suppose you prompt your model “Write a Python function to merge two sorted lists” 64 times at temperature 0.8. You run each output through a test suite as your quality metric. Maybe 40 pass all tests. You take the 10 cleanest, most readable passing solutions and add them to your SFT dataset. You’ve just used the model’s own competence to build training data better than what most human annotators would produce for a coding task.

It sounds like RL, but it’s just using the same pieces to filter the demonstrations that the model should see in SFT. The same graders like reward models, verifiers, or LLM-as-judges are used to curate SFT data.

Rejection sampling is also surprisingly effective and therefore popular; for example it was described early on in Meta’s Llama 2 post-training pipeline. The model already can produce great responses, but as you’ve probably noticed, it just doesn’t do so reliably. By filtering for its best outputs and training, you raise its average toward its ceiling. Instead of acting as the average developer, it’s nudged to act as an expert developer. Rejection sampling scales well because generation is cheap relative to human annotation. The main limitation is that you’re still bounded by what the model can produce at sample time. If it can’t generate a correct proof in any of 100 attempts, no amount of filtering will help.

But SFT has limitations. It only teaches the model what to do. You’re presenting ideal behavior but never showing it what “bad” looks like. As a result, the model could still produce problematic outputs on prompts that weren’t well-represented during training.

The SFT model is also prone to “mode averaging” when the training data sends mixed signals. For example, if half your golden retriever demonstrations sound like an encyclopedia (“The Golden Retriever (Canis lupus familiaris) is a large-sized breed of gun dog…”) and the other half sound really casual (“Golden retrievers? They’re basically furry happiness machines”), the model won’t learn to pick the right tone for each context. It’ll blend them into an awkward middle: “The Golden Retriever is basically a large-sized happiness machine of the gun dog variety.” Neither formal nor casual, which comes off as weird and not the right response style.

Why frontier models use both

RL seems all-powerful. Why not use it alone? This was a research question pursued by DeepSeek’s team when training DeepSeek R1-Zero. Up until then, the base models were so bad that doing RL was pointless on them and you needed to do SFT. This model demonstrated that RL applied directly to their relatively strong pretrained model can produce powerful reasoning ability without any SFT.

However, the model still had serious usability problems. For example, it would mix languages (e.g., English with Mandarin), so it was difficult to use for most people. It could reason, but it wasn’t practical to use.

RL’s main ceiling after the model has been trained is usability. During training, its ceiling is stability. Research on new methods are continually trying to find ways to do RL post-training more stably.

SFT, on the other hand, has the opposite problem. It’s been used alone for many years and has reached maturity to some degree. InstructGPT made the model capable of instruction-following, and became the foundational approach for ChatGPT to handle multiturn dialogue and thus conversation. However, while SFT gets good, reliable results, it’s typically not enough to push performance at the frontier to reach superhuman performance on important tasks.

Here’s what it means for you: If you’re doing post-training on your own, and you want your model to behave a certain way and you don’t care about novel frontier performance, SFT will get the job done.

In contrast, RL can teach a model to reason through novel problems it hasn’t seen during training, because the reward signal evaluates the outcome rather than the exact token-by-token process like in SFT. RL can surface rare but important behaviors that might not appear frequently enough in any SFT dataset.

On scaling data, RL can improve a model’s performance on a task as long as the reward signal is accurate, without needing to collect additional human-written examples. However, it’s important to note that on some tasks, it’s easier to scale SFT examples, and on others, it’s easier to scale via RL. For example, RL scales more easily on math problems. You can generate an unlimited number of math problems programmatically, and a verifier can check whether the answer is correct with certainty. You wouldn’t need to hire a mathematician to write out ideal solutions. The model attempts problems, gets told right or wrong, and improves.

Safety is another area where RL stands out. It’s relatively easy to write a few hundred examples of a model declining harmful requests in your SFT dataset. But the space of ways a user might try to get harmful or inappropriate content is broad, creative, and ever-changing. RL allows the model to be trained against adversarial prompts, where it practices handling tricky edge cases and gets rewarded for handling them well. This is much harder to achieve with static demonstration data alone.

Meanwhile, SFT scales more easily on writing in a specific brand voice. If you want the model to respond with your company’s brand voice. It might be hard to write a reward function that captures “sounds like our brand.” But the company could have tens of thousands of real support transcripts that already demonstrate the voice. You can curate these, transforming them into prompt-response pairs. The data already exists at a decent scale, while the reward signal would be hard to get right.

When a frontier lab wants to add support for a new feature, for example calling MCPs or calling subagents, the first step is almost always to create a small amount of SFT data demonstrating that capability. The next step is creating a reward function and RL environment that can match it.

Far more data and thus compute are dedicated to RL than SFT, but SFT offers good warm starts for the model and those examples are critical to getting the model into a stable place for subsequent RL.

The combination of both is ultimately what makes modern frontier models as capable as they are. Neither alone is sufficient.

A standard post-training pipeline uses SFT and RL as complementary stages that build on each other. It might look like this:

1. Pretraining produces a foundation model with broad knowledge.
2. SFT takes that foundation model and teaches it basic behaviors: how to have a conversation, follow instructions, use a helpful tone on a range of different tasks, etc.
3. RL takes the SFT model checkpoint and refines it further. Using reward signals from human preferences, programmatic verifiers, or AI judges, RL gets the model to be more consistently helpful, less likely to produce harmful content, and better at complex tasks like reasoning.

Some teams also iterate between several stages of SFT and RL: SFT, then RL, then more SFT on new data, then more RL. The first couple stages could be on reasoning for verifiable tasks like math and code where the data and reward signals (verifiers) are constructed differently, whereas the second could be on messier general reasoning over all tasks, which would involve training reward models that encode human feedback as preferences.

This iterative refinement can help with checkpointing quality at different stages and handing things off to different teams, though it adds complexity to the pipeline. Not surprisingly, the quality of each previous stage directly affects how well subsequent stages can go.