DPO, dissected

What exactly is a pretrained language model? Let's try to understand. In a simple way. A language model starts life reading a huge pile of internet text, chopped into small pieces called tokens.

And it has one job only. Guess the next token. That is next-token prediction. And notice what is missing.

Nobody told the model what a good answer looks like. Only what text usually comes next. This training makes the model genuinely capable. Broad world knowledge, even some reasoning.

But precise control over its behavior? Nothing in training ever asked for that. Here is a concrete example. Imagine a misconception that fifty percent of people believe.

A model trained to imitate typical text may assert it in about fifty percent of its answers. We want it aware of the misconception, yet never stating it as fact. Same story with code. To fix common bugs, a model must know them well.

But asked to write code, it should produce the best code it has seen, not the average. So here is the key idea. The capability is already inside. The problem is selection.

Out of everything the model could say, how do we pull out what we actually want? That selection job has a name: alignment. The paper this course dissects, Direct Preference Optimization, DPO for short, is one answer. We will earn it step by step.

So remember this. A pretrained model is brilliant autocomplete, not an assistant. It predicts likely text, not desired text. The rest of this course is about closing that gap.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is supervised fine-tuning? Let's try to understand. In a simple way. Last time we saw the problem.

A pretrained model completes text; it does not follow instructions. The obvious fix: show it examples of the behavior you want. So you collect prompts with high quality answers, written by people. Then you keep doing next token prediction, but only on these examples.

That is supervised fine-tuning, SFT for short. The result is called the SFT model. And it works. Ask a question, and it now answers instead of rambling on.

Every pipeline in this course, DPO included, starts here. But here is the ceiling. SFT is pure imitation. Every example in the dataset is treated as equally correct.

The training has no dial for quality. Say two demonstrations answer the same question. One is brilliant, one is just okay. SFT pushes the model toward both with the same force.

There is no way to say, this one is better. Second ceiling. Someone has to write those perfect answers, and good demonstrations are slow and expensive to produce. Worse, the model is capped by the skill of whoever writes them.

Now notice something. Writing a great explanation of, say, quantum computing is hard. But reading two explanations and pointing at the better one? Almost anyone can do that in seconds.

That gap, between authoring and judging, is the door the rest of this course walks through. Next time: turning those quick judgments into training data. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is preference data? Let's try to understand. In a simple way. Last lesson ended with a gap.

Supervised fine-tuning treats every example as equally correct. So where does a better or worse signal come from? The obvious idea first. Show a human one answer, and ask for a score.

Say one to ten. Now we have numbers. Problem solved, right? Not really.

Your seven is my four. And even one person drifts. After ten harsh answers, a mediocre one suddenly feels like an eight. Absolute scores are noisy and inconsistent.

Here is the fix. Show two answers side by side, and ask one question. Which is better? People are remarkably consistent at that, even when their scales disagree.

So each data point becomes a triple. The prompt, the winner, and the loser. This is called a preference pair, and it is the fuel for everything that follows. Where do the two answers come from?

In the paper's setup, the fine-tuned model itself writes both. Sample two answers to the same prompt, then let a human pick. And remember the cost point from last lesson. Writing a perfect answer is hard.

Picking the better of two takes seconds. So this data scales. Now the key assumption. Behind every choice, we imagine a hidden quality score in the human's head.

We never see it. Each comparison is one noisy glimpse of it. So, preferences trade broken scores for stable choices. Next we need math that turns those choices back into numbers.

That is the next lesson. Quick check now. One question is coming up. Let's see if it clicked.

How does the Bradley-Terry model work? Let's try to understand. In a simple way. Last lesson gave us preference pairs: winner and loser.

We assumed each response has a hidden quality score. But scores are invisible. Comparisons are all we see. Bradley and Terry faced this in nineteen fifty two.

Their insight: a stronger player does not always beat a weaker one. They just win more often. Strength shows as probability. So here is the recipe.

Take e, a fixed number near two point seven, to the power of each score. That makes both numbers positive. Then the winner's chance is simply its share of the total. Say one response scores two, the other zero.

Exponentiated, about seven point four against one. The stronger response wins about eighty eight percent of the time. Equal scores? One half.

Now the property everything turns on. Add ten to both scores. The chance does not move at all. Anything added to both sides cancels.

Only the gap between the scores matters. So the whole formula collapses into one curve, the logistic function of the gap. Feed it any number, it squashes it into a probability between zero and one. Gap zero gives one half.

Why probabilities, though? Because human judges are noisy. With a small gap, the weaker answer still wins sometimes. Each comparison is just a noisy glimpse of the hidden scores underneath.

Hold on to that cancellation. Anything shared by both responses vanishes from the math. The entire DPO trick, coming later, leans on exactly that property. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is a reward model? Let's try to understand. In a simple way. Last lesson turned score gaps into win probabilities.

But nothing so far actually produces the scores. We need a judge that can score any new response on its own. Enter the reward model. A separate neural network that reads the prompt and one response, and outputs a single number.

Higher means humans would like it more. And you don't build it from scratch. Take the SFT model, swap its next-token layer for a tiny head that outputs one number. So the judge starts out already understanding language.

Training uses the same preference triples. Score the winner, score the loser, subtract. Feed that gap into the squashing function from last lesson. Out comes a predicted chance that the winner wins.

Try it. Say the judge scores the winner two and the loser five. The gap is minus three, so the predicted chance is tiny, around five percent. The loss punishes that hard.

And that's the whole loss. Push each winner's score above its loser's. Badly ordered pairs get a strong correction. Pairs already ranked right barely move the weights.

One quirk. Add ten to both scores and nothing changes; only the gap enters the loss. Scores have no absolute meaning, so rewards are often normalized to average zero per prompt. Hold onto this quirk.

So we now hold a trained judge that scores any response, instantly, no human needed. Next time: how to use it to actually teach the model. Quick check now. One question is coming up.

Let's see if it clicked.

How does reinforcement learning against the judge work? Let's try to understand. In a simple way. Last lesson we built a judge that scores any response.

Now the payoff. We use that score to improve the model itself. First, reinforcement learning renames things. The model is now the policy, the thing choosing what to do next.

The prompt is the situation, the response is the action, the score is the reward. The loop goes like this. The policy writes a response, the judge scores it, and we nudge the numbers inside the model. Those tokens get more likely if the score was good, less likely if bad.

That nudge is called the policy gradient. Repeat it thousands of times, and the model drifts toward answers the judge loves. Say the prompt asks for a polite refusal. A kind, clear reply scores high, so its tokens get boosted.

A rude one scores low, its tokens get pushed down. The score must be about what this model, right now, actually writes. So training keeps generating fresh responses, live in the loop. Why this roundabout route?

Picking a token is a jump, not a smooth dial, and ordinary training can only turn smooth dials. So we score finished text, then nudge. One specific algorithm runs this loop in practice. PPO, short for Proximal Policy Optimization.

Remember that name. DPO gets measured against it. So this whole stage is a policy chasing a judge's score. Chase an imperfect judge too hard and things get weird.

Next time. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the KL leash? Let's try to understand. In a simple way. Last lesson ended with a loop.

The model writes, the judge scores, the weights nudge toward higher scores. Sounds perfect. It is not. Here is the thing.

That judge is only an approximation of human taste, fitted on a limited pile of comparisons. It has blind spots. Guaranteed. Push hard enough on any approximation and you find where it breaks.

The model starts writing text that scores high without being good. The field calls this reward hacking. Picture a judge with a soft spot for long, confident answers. The model will find it, and every reply becomes padded, overconfident sludge.

High score, worse model. So the training recipe adds a leash. Keep a frozen copy of the SFT model, and call it the reference model. It never trains.

It just watches. Then measure the drift. The KL distance compares two sets of probabilities. Zero when the trained model still matches the reference, growing as they pull apart.

Now the real objective. Maximize the average score, minus beta times the KL distance. Chase reward, but pay a fine for drifting away from the reference. Beta sets the fine.

Large beta, short leash, tiny movement. Small beta, more freedom, more drift. And no, beta is not a learning rate. The leash keeps the model where the judge is still accurate, and it stops collapse onto one high-scoring reply.

Remember this objective. DPO will solve exactly this one. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is the real cost of RLHF? Let's try to understand. In a simple way. Look at the pipeline so far: supervised fine-tuning, then train the judge, then reinforcement learning against it.

The field calls this RLHF, reinforcement learning from human feedback. Three separate stages, each able to fail on its own. And count the networks. The model being trained, the frozen reference, and the reward model.

Three large models in play, and typical implementations add helper networks on top. Now the slow part. During the reinforcement learning stage the model writes fresh responses live, inside the training loop. Every step: generate, score, nudge.

Generating text is expensive. This has a name: on-policy sampling. The scores must be about what this model, right now, actually writes. Yesterday's samples describe yesterday's model, so old data can't be reused.

Then the signal itself. A whole response earns one single score, and from that one number the model must work out which token choices deserve credit. Thin, noisy feedback. The paper points at this.

Raw scores make the nudges swing wildly, so learning goes unstable. The standard patch: subtract a baseline, a typical score, often from one human-written answer. So add it up. Three stages.

Three or more networks. Live generation inside every step. Noisy, patched gradients, and fiddly tuning. All that machinery to optimize the leash objective from last lesson.

Here is the thing though. That leash objective is a clean math problem, and clean math problems sometimes have exact answers. Next lesson, we solve it on paper. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the closed-form optimum of the RLHF objective? Let's try to understand. In a simple way. Recall the goal.

Get the highest score from the judge, while paying a fine for drifting from the frozen reference. Here is the surprise. That tug of war has an exact answer, a formula you can write down. That is what closed form means.

The recipe: take the reference model's probability for a response. Multiply by the exponential of the score divided by beta. Then divide by a total, so probabilities add up to one. Try it.

Two equally likely responses. One scores plus one, one minus one, beta is one. The winner's weight grows about two point seven times, the loser's shrinks the same way. After dividing, the winner holds about eighty eight percent.

Beta sets how violent the boost is. Shrink beta to a tenth, and scores get multiplied by ten inside the exponential. Tiny gaps become landslides. Grow beta, and the model barely moves.

This formula is not DPO's invention. It is a standard result in the reinforcement learning literature. What is new is the use the paper makes of it. Now the catch.

That divide-by-a-total step is called the partition function, or the normalizer. It sums over every response the model could possibly write. An astronomical count. So the formula is exact, and completely useless to compute.

For now. Last thing. The formula holds for any reward, even the true unknown one behind human preferences. Hold that thought.

Next lesson, we flip it inside out. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the flip at the heart of DPO? Let's try to understand. In a simple way. Last lesson ended on a cliffhanger.

An exact formula for the best policy, blocked by the normalizer, that sum over every possible response. Exact, and useless. So here is the trick. Don't solve that equation for the policy.

Flip it, and solve for the reward instead. Same equation, read backwards. Take logarithms, the undo button for exponentials, and rearrange. A response's reward is beta times the log of its probability under the optimal policy over its probability under the reference.

Plus a leftover term. In plain words: the score is how much more likely that model is to write the response than the frozen copy. The judgment lives in the ratio. Try numbers.

Four times likelier than under the reference? Log positive, score positive. Half as likely? Log negative, score negative.

Now the leftover. It's beta times the log of the normalizer, and it depends only on the prompt, never the response. And the paper proves it: rewards that differ only by a prompt-only shift give the same preferences and the same optimal policy. That leftover is harmless.

That is the paper's title. Your language model is secretly a reward model. Any model, read against a reference, defines a score. The paper calls it the implicit reward.

And nothing is lost. The paper proves every reward consistent with Bradley-Terry can be rewritten this way. But the normalizer is still there, uncomputable. Next lesson, it vanishes.

Quick check now. One question is coming up. Let's see if it clicked.

How does the DPO loss work? Let's try to understand. In a simple way. Last lesson flipped the formula: a response's reward is beta times the log ratio of policy to reference, plus a partition function term.

Now plug that into Bradley-Terry, which only uses the difference of two scores. The partition function depends only on the prompt, so it cancels. Exactly. What remains is computable.

Fit it to the preference pairs, making each human choice as probable as possible, and out falls the DPO loss. Term one: the winner's lift. How much more likely is our model to write the winning answer than the frozen reference is? Log it, scale by beta.

Term two: the same lift for the loser. Subtract loser from winner, squash the gap with the logistic function, and average over the dataset. Now recall lesson five's judge loss. Same shape, line for line.

Only the score slot changed: a separate trained network before, a ratio of two probabilities now. The gradient pushes the winner's probability up and the loser's down, weighted by how wrongly the implicit reward currently ranks each pair. Pairs already ordered right contribute almost nothing. And beta?

It's the same beta from the KL leash, arriving by algebra. Bigger beta makes a small lift count as a big score. A tighter tie to the reference. Beta is not a learning rate, the step size of each weight nudge.

That was one millionth in the paper, while beta was zero point one, or zero point five for summarization. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is DPO's report card? Let's try to understand. In a simple way. The paper tested DPO on three jobs: steering sentiment in movie reviews, summarizing Reddit posts, and single turn dialogue.

Largest model: six billion parameters. On summarization, GPT four was the judge. DPO beat human written summaries about sixty one percent of the time. PPO's best was fifty seven.

Turn up sampling randomness, called temperature, and PPO fell off sharply while DPO held steady. On dialogue, it was the only efficient method to beat the dataset's own preferred answers. Trained on Reddit, then tested on news articles it never saw, DPO still beat PPO. The authors call that preliminary, though.

Now the caveat. DPO trains on a fixed dataset, never sampling from the model. That's the selling point. But those preference pairs came from an older model.

Later work probed this. As training moves the policy away from that data, the labels fit it less, and a well tuned PPO can still beat DPO. A family followed. IPO drops the Bradley Terry assumption to curb overfitting.

KTO needs no pairs, just answers labeled desirable or not. ORPO even deletes the reference model. And GRPO swung back to online reinforcement learning, simplifying PPO by scoring a group of answers against the group's own average. Reasoning models were later trained this way.

So here's the honest ending. DPO didn't settle the argument, it flipped the burden of proof. Reinforcement learning now must earn its cost, and DPO remains standard for open weight models. Quick check now.

One question is coming up. Let's see if it clicked.