InstructGPT, dissected

What exactly is next-token prediction? Let's try to understand. In a simple way. A pretrained GPT-3 has one job: read internet text and guess the next token, the next small chunk of text.

Nothing else was ever scored. So an instruction is not heard as a request. The model sees a document that stops mid-way, and continues it in the most likely direction. You type: give me three tips for sleeping better.

Online, a line like that often sits in a list of headings. So the likeliest continuation is another heading: give me three tips for waking early. Likely text, not an answer. The paper names three failures.

Untruthful: a confident made-up fact can be the likeliest text. Toxic: the internet holds toxic text, so continuing it is likely. Unhelpful: our list of headings. And making the model bigger does not fix this.

A bigger model is a better predictor of internet text, and the internet was never written to obey you. The authors call this a misaligned objective. Next-token prediction is only a proxy, a stand-in, for what the user actually wants. The paper borrows three words for what we want.

Helpful: it does the task. Honest: it does not assert what it has no basis for. Harmless: it does not hurt people. None of these appear in the training objective.

Every model in this paper keeps the GPT-3 architecture, at one point three, six, or one hundred seventy five billion parameters. Only the training target changes. How, is the rest of this course. Quick check now.

One question is coming up. Let's see if it clicked.

How does few-shot prompting work? Let's try to understand. In a simple way. Last time we saw the problem.

A pretrained model only predicts the next token, so a bare instruction gets whatever the internet would write next. Before anyone retrained the model, people found a workaround. Do not just ask. Show.

Put a few finished examples before your request, and the model continues the pattern. Want a story about a wise frog? Paste two frog stories first, then start a third. The most likely next tokens are now another frog story.

The paper lists three shapes a task can take. A direct instruction. A few examples, then a request. Or just the opening of what you want.

The authors took this seriously. For their prompted baseline, two of them held a one-hour prefix-finding competition, hunting for the few-shot prefix that best pushed GPT-3 into instruction-following mode. And it works, to a point. Same weights, zero training, and the outputs land more on task.

The prefix does all the work. But the prefix has a price. It sits inside the model's input on every single call, and it can only nudge the style of what comes next. Here is the ceiling.

The model is still predicting text. Examples change what is likely, not what is wanted. It has no notion of helpful, honest or harmless. The paper's framing is blunt: the language modeling objective is misaligned, and a prompt only coaxes it.

To change what the model is trying to do, change the training. Next lesson. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is learning from demonstrations? Let's try to understand. In a simple way. Last time, prompts only coaxed a model built to predict the next token.

Now the team changes the model itself, and the first move is surprisingly plain. Take a real prompt, say, write a story about a wise frog. A hired labeler writes the ideal answer by hand. Prompt and answer together form one demonstration.

Training uses nothing new. The same next-token loss from pretraining, only now the text is the labeler's answer after the prompt. The paper calls this supervised fine-tuning, SFT for short: the model learns to imitate what a human wrote. The dataset is small.

About thirteen thousand prompts, most written by labelers, some sent in by early users. That is tiny next to the internet. They train for sixteen epochs, sixteen passes over those same demonstrations. That is a lot of repetition.

Here is the odd part. On held-out prompts, the loss starts getting worse after just one pass. By the textbook, that is overfitting. Stop.

Yet human ratings of the answers keep climbing across all sixteen passes. So the team kept training anyway, trusting the ratings over the loss. The lesson: the loss measures how well you predict a labeler's exact words. People judge whether the answer is good.

Those are different questions. And here is the ceiling. Imitation can only copy what a human wrote. If the labeler's answer is average, the model learns average.

It never learns which of its own answers is better. Next lesson. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is comparison data? Let's try to understand. In a simple way. Last lesson ended at a wall: imitation only copies what a human wrote.

To go further, the model needs to hear how good its own answers are. The obvious way is a score. Read one answer, give it seven out of ten. But your seven is my five.

People rarely agree on how good something is. Put two answers side by side and ask which is better. Now people tend to agree more. Comparing is easier than grading.

That is the paper's bet. The task: a labeler gets one prompt and several model answers, between four and nine. They rank the whole set from best to worst. First each answer gets a score from one to seven, plus tick boxes: did it follow the instruction, did it invent facts.

Then the ranking, with ties allowed for look-alikes. One ranking is worth many comparisons. Rank four answers, six pairs. Rank nine, thirty six pairs.

One session yields a pile of which-is-better data. How much do labelers agree? The training team, about seventy two point six percent of the time. A held-out group, never used for training data, seventy seven point three.

Read that carefully. Even careful humans disagree roughly one time in four. The preferences are noisy, and anything learned from them cannot be surer than the people behind them. So we have rankings for about thirty three thousand prompts, and nothing trained on them yet.

Next lesson, this pile becomes a judge. Quick check now. One question is coming up. Let's see if it clicked.

How does training a judge work? Let's try to understand. In a simple way. Labelers ranked several answers to one prompt.

Now a model learns to predict those rankings: the reward model. Think of it as a judge. Start from the supervised model. Its last layer gives a score for every possible next token.

Swap it for a layer that outputs one number: the reward for a prompt and its response. Training uses pairs. Score the preferred answer, score the rejected one, subtract. The judge is pushed to make that gap large and positive.

That gap has a meaning: the log odds that a labeler prefers the first answer. Zero means a coin flip. Bigger means a more confident bet. Here is a trap.

Rank six answers to one prompt and you get fifteen pairs. Shuffle every pair across the dataset, and each answer gets up to five separate updates. The judge overfits in one pass. The fix: keep all pairs from one prompt together as one batch element.

Each answer is scored only once, and validation accuracy improved a lot. Only differences matter, so adding a constant to every reward changes nothing. They add a bias so human demonstrations score zero on average. How big is the judge?

Six billion parameters, whatever the size of the model it scores, even the largest. It saves compute, and a bigger judge trained unstably. One epoch only; more overfit quickly. So now a model scores any answer with one number.

It has not trained anything yet. That comes next. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is reinforcement learning? Let's try to understand. In a simple way. We now have a judge: a six billion model that reads a prompt and a response and returns one number.

Today the judge finally goes to work. First word: policy. The policy is the model being trained, starting as the SFT model. Given a prompt, it writes a response token by token.

That writing is its action. Second word: reward. Hand the prompt and finished response to the judge. It returns one scalar, say plus one point two.

Higher means labelers would probably prefer it. Third word: episode. Here it is tiny. One prompt appears.

The policy writes one whole response. The judge scores it once. Episode over. Nothing carries to the next prompt.

The paper calls this a bandit environment. Think of a slot machine: one pull, one payout, no board, no next move. A whole response is one pull. Now the learning.

Responses that scored high are made more likely. Responses that scored low, less likely. The policy's token probabilities drift toward whatever the judge rewards. The optimizer doing the nudging is called PPO.

Its job here: push the policy toward higher scores in small clipped steps, so one batch cannot yank it far. At scale: about two hundred fifty six thousand episodes over roughly thirty one thousand distinct prompts, five hundred twelve episodes per batch. So the judge never writes. It only scores.

The policy writes, and chases the number. What happens when it chases too hard? Next lesson. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the KL leash? Let's try to understand. In a simple way. Last time, the policy chased the judge's score.

But the judge is a trained model, and trained models have blind spots. Chase hard enough and the policy finds them. Picture a student who learns what the grader loves. Long confident answers, a favorite phrase.

Soon the student writes for the grader, not the reader. That is over-optimization of the reward model. The fix. Leash the policy to the SFT model it started from.

Measure how far its probabilities drifted, a distance called KL divergence, and charge for it. Token by token. Take the policy's probability for the word it chose, divide by the SFT model's probability, take the log. Same probability, zero charge.

Ten times more likely, about two point three. Multiply by beta, set to zero point zero two. So a ten times drift costs about zero point zero five at that token. Tiny, but it adds up.

Why fold it into the reward, not a separate loss? Then the judge's score and the drift charge share one currency. The policy chases the score after the charge. That changes what wins.

A response the judge adores but SFT finds bizarre gets docked. One the judge likes and SFT finds natural survives. Notice the anchor. It is the SFT model, not raw pretrained GPT-3.

The leash pulls toward answers that already follow instructions. One thing a leash cannot do: hand back skills the policy loses on the way. That needs a different fix. Quick check now.

One question is coming up. Let's see if it clicked.

How does the full InstructGPT pipeline work? Let's try to understand. In a simple way. Over the last lessons we met three steps one at a time: demonstrations, a judge, reinforcement learning on a leash.

Now we lay them end to end. Every step starts with prompts, from two sources. Labelers wrote some. The rest came from real users typing into an early version of the model, with private details filtered out.

Three piles. About thirteen thousand prompts for demonstrations, mostly labeler written. Thirty three thousand for rankings, mostly from users. Thirty one thousand for reinforcement learning, all from users, with no human labels.

Why no labels on the third pile? Because by then the judge exists. A prompt goes in, the policy answers, the judge scores it. The human work was spent earlier.

Now the hand-offs. The supervised model seeds two things. The policy starts as a copy of it. The judge starts as another copy, its word-predicting head swapped for a single number.

The judge seeds one more piece. PPO keeps a helper, the value function, which guesses a response's reward before the judge speaks. That helper starts as a copy of the judge. One more link.

Last lesson's leash is measured against the supervised model too. So one model is the starting point, the judge's backbone and the anchor. Now the names. SFT is step one alone.

PPO is the leashed policy. PPO-ptx adds a pretraining ingredient we meet next lesson. And when the paper says InstructGPT, it means PPO-ptx. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the alignment tax? Let's try to understand. In a simple way. Last lesson we assembled the pipeline.

The PPO model wins on labeler preference. But something was quietly lost. The paper calls it the alignment tax. A public benchmark is a fixed test set anyone can download.

The paper checked four: SQuAD and DROP for reading questions, HellaSwag for finishing a sentence, and French to English translation. At one hundred seventy five billion, the base model scored about sixty four on SQuAD. Plain PPO scored about forty four. Twenty points gone.

Translation fell from about thirty nine to about twenty four. The pattern held: sharper at pleasing the judge, worse at old skills. First idea: tighten the KL leash from lesson seven. Pull the policy closer to the SFT model, and maybe the old skills stay.

It fails. Even at beta two, one hundred times the default, DROP and SQuAD never fully recover, and the judge's validation score drops sharply. Preference lost, skills not bought back. Second idea: during PPO, keep practicing next token prediction on ordinary pretraining text.

Each update pushes toward the judge and toward the old objective at once. That pretraining gradient is scaled by twenty seven point eight, with eight pretraining examples for every RL episode. One value worked across all model sizes. This is PPO-ptx.

SQuAD climbs back to about sixty, translation to about thirty four, HellaSwag even beats the base model. DROP still lags. And labeler preference barely changes. Quick check now.

One question is coming up. Let's see if it clicked.

How does a small model beat a big one? Let's try to understand. In a simple way. Every result today is a win rate: labelers see two answers to a real prompt and pick one.

The yardstick is the one hundred seventy five billion SFT model, since it sits mid-pack. Against plain GPT-3 at the same size, InstructGPT wins about eighty five percent of the time. Against GPT-3 with a hunted few-shot prefix, still seventy one. The ladder never reorders: plain GPT-3, prompted GPT-3, SFT, then PPO on top.

Mixing pretraining back in barely moves preference. Now the headline. The one point three billion InstructGPT, one hundred times smaller, is preferred over the one hundred seventy five billion GPT-3. Same architecture; only the human data differs.

Did it just learn the tastes of its forty labelers? Held-out labelers, who wrote no training data, ranked again. Same order, same wide gap. Sharper test: split labelers into five groups, train a judge on four, test on the fifth.

Accuracy slips from seventy two point four to sixty nine point six. A dip, not a collapse. What about FLAN and T0, GPT-3 tuned on about a million public examples each? Against the SFT yardstick they win under thirty percent.

InstructGPT wins seventy three. The paper's reading: public datasets lean on classification and question answering, about eighteen percent of real usage. Open generation and brainstorming, about fifty seven percent, is what people ask. One caution.

Preferred means labelers liked it more. Truthful, or less toxic, is a separate question. Next lesson. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is InstructGPT's safety report card? Let's try to understand. In a simple way. Lesson one named three targets: helpful, honest, harmless.

Lesson ten showed helpful moved a lot. Now the other two, reported honestly. The paper cannot see what a model believes, so it measures truthfulness: are the statements about the world actually true. Two tests.

First, TruthfulQA, a quiz of questions people commonly get wrong. The PPO models gave truthful and informative answers about twice as often as GPT three, with no instruction to be truthful. One exception. The one point three billion PPO-ptx model came out slightly worse than plain GPT three of the same size.

The headline does not hold at every scale. Second, closed-domain tasks, where the answer must come from the given text. Labelers flagged made-up facts in twenty one percent of InstructGPT outputs, against forty one percent for GPT three. Now harm.

On prompts designed to provoke toxic text, InstructGPT produced about twenty five percent fewer toxic outputs. But only when the prompt asked it to be respectful. Remove that instruction and the advantage disappears. And when explicitly asked to be toxic, InstructGPT is much more toxic than GPT three.

It follows instructions, in both directions. Bias, measured by how evenly the model splits between a stereotyped sentence and its opposite, did not improve. Told to be respectful, it actually became more biased. So the scorecard: truthfulness up, hallucinations down, toxicity down only on request, bias unchanged.

Alignment followed the instruction it was given, nothing more. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the fine print of InstructGPT? Let's try to understand. In a simple way. Every ranking came from about forty contractors, mostly English speakers in the United States or Southeast Asia.

The model learned their taste, and the authors say so. Most comparisons were labeled by one person, to save cost. Labelers agree about seventy three percent of the time, so a second labeler might flip roughly one pair in four. The authors name the greatest limitation: in most cases the model obeys the instruction even when obeying could cause real harm.

Refusals were left for future versions. Pretraining GPT-3 cost three thousand six hundred forty petaflops per second days, a compute unit. PPO-ptx cost sixty. Sixty times cheaper, for a gain the authors rank above a hundred-fold scale-up.

The recipe was borrowed. Christiano and colleagues in twenty seventeen learned rewards from human comparisons on games and simulated robots. Stiennon and colleagues in twenty twenty trained a summarizer that way. Here it widened to instructions.

Then Constitutional AI swapped human harmlessness labels for AI feedback guided by written principles. DPO dropped the judge and the RL loop, turning preference pairs into a classification loss. Llama Two's chat models used human feedback too. Their open list: hunt adversarially for worst-case behavior, make refusals configurable, try optimizers beyond PPO, collect richer feedback like edits, and condition models on particular groups' preferences.

The verdict. A small model tuned to forty people's preferences beat one a hundred times larger. Borrowed recipe, few judges, and whose values count stays open. Quick check now.

One question is coming up. Let's see if it clicked.