Post-training: teaching manners and skills

What exactly is post-training? Let's try to understand. In a simple way. Pretraining gave us one thing.

A machine that guesses the next word. You feed it text, and it continues that text. That was the only job. But nobody ever trained it to help you.

Its whole goal was to match text on the internet. Not to answer the person typing. So the training runs in two stages. First pretraining, where it reads and predicts.

Then a second stage on top. That second stage is post-training. Post-training does not build a new model. It takes the pretrained weights and keeps training them, just with different data and a different goal.

And here is the key. It is not pouring in new facts. The facts are already in there. It is changing which behavior comes out.

The pretrained model can already write a helpful answer. It can just as easily ramble instead. Post-training makes the helpful one its default. And it is small.

Pretraining chews through trillions of words for months. Post-training runs on a much smaller pile, for a sliver of the compute. Post-training is not one trick. It is a family of steps.

First showing it good answers, then tuning it on what people prefer. This is why model pages list two versions. A base model, and an instruct model. Same size, same knowledge.

One of them went through post-training. So, post-training is the second stage that turns a text predictor into something that follows instructions. Same knowledge. New habits.

Quick check now. One question is coming up. Let's see if it clicked.

How does asking a base model a question work? Let's try to understand. In a simple way. You type a real question into a base model.

What comes back is not an answer. It is more questions, or a wall of loose text. Here is why. The base model does not see a question.

It sees a piece of text that stopped early. Its only job is to continue it. So it asks itself one thing. What usually comes next, on the internet, after words like these?

Then it writes that. On the internet, a lone question usually sits on a page with more questions. A quiz. A forum thread.

So it gives you the list. There is no you in its world. It does not know a person is waiting. It is finishing a document, not talking to anyone.

Now the real mechanism. Training scored one thing. Did it predict the next word? Helpfulness was never in that score.

So nothing pushed it there. It also copies your style, because matching text is the goal. A sloppy prompt gets sloppy text back. And it does not know where to stop.

Real documents keep going after an answer, so it keeps going too. Straight past the point. Proof it can. Shape the page as a question and answer transcript, and a real answer appears.

The skill was there. The default was not. So, base models are not bad at helping. They were never aiming at it.

They aim at likely text, and that is rarely an answer. Quick check now. One question is coming up. Let's see if it clicked.

How does teaching a model with question and answer pairs work? Let's try to understand. In a simple way. The fix sounds almost too simple.

If you want answers, show it answers. Thousands of them, written out by hand. One example is just a pair. A question someone might ask, and a good answer sitting right next to it.

That is the whole example. Now the trick. Glue the two together into one document. Question, then answer.

Suddenly it is only text again, and predicting text is the one thing this model already does. But it is graded on the second half alone. The question is context, never scored. Every word of the answer is scored.

That is the real difference. So the loop is the familiar one. Cover the next word of the answer, let it guess, measure how wrong it was, nudge the weights. Then the next word.

This step has a name. Supervised fine-tuning. Supervised, because the right answer was handed over, not discovered. And it is not memorizing those answers.

Nobody asks the exact same question later. Across thousands of pairs it picks up the shape. A question arrives, an answer follows. The pile stays small.

Tens of thousands of pairs, not trillions of words. Hours of training, not months. One catch. It only ever sees good answers.

It is shown what to copy, and never once what to avoid. So, pairs turn the same next word machine into an assistant. Same loop. New examples.

Graded only on the answer half. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is an assistant persona? Let's try to understand. In a simple way. Every conversation, the same character shows up.

Same steady tone. Same way of saying it is not sure. That voice is the persona. But the base model has no voice of its own.

It read the internet, so it can play a pirate, a lawyer, an angry commenter. Every voice, no default. Fine-tuning does not invent a voice. It picks one already in there, and makes it the default.

How? Every answer in the pile was written to one style guide. Be helpful. Admit uncertainty.

Do not lecture. Thousands of answers, one set of rules. That consistency is the persona. Now the real mechanism.

Every turn is wrapped in special markers. A user tag, then an assistant tag. The text after that tag is always the same voice. Ask who it is, and it answers.

Not because it knows. Because someone wrote that answer, in that voice, and trained it in. Identity is data. The persona covers refusals too.

The same calm wording every time, because the pile held refusal examples, in that same voice. There is a lever. A system tag sits above the conversation. Training used many different system prompts, so one line from you bends the persona.

And the other voices never left. Fine-tuning moved the default. It deleted nothing. That is why role-play works, and why jailbreaks work.

So, the persona is a default voice. One consistent pile of answers, behind an assistant tag. Not a self. A habit.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is so hard about writing good answers? Let's try to understand. In a simple way. Fine-tuning needs a pile of question and answer pairs.

Those answers were not found anywhere. Someone sat down and typed every one, from a blank page. That is slow. One careful answer takes a skilled writer twenty minutes.

Multiply by fifty thousand. The bill is people, not chips. Worse, most questions have no answer key. Comfort a friend who lost a job.

A thousand replies are good, and none is the right one. Hand that question to two good writers, and you get two different answers. Both fine. Only one can go in the pile.

Here is the real mechanism. Training holds up that one chosen answer, and compares the model word by word against it. One string is the target. So a perfect answer in other words gets pushed down, just like a bad one.

The loss cannot tell different from worse. It only checks, did you say my word. Some questions need an expert. A kidney question needs a doctor.

A code question needs an engineer. The writers get rarer, the cost climbs. Then the ceiling. The model is copying people, so it lands near the writers it copied.

Copying can match them, never pass them. And every new topic needs fresh writing. Nobody can hand write an answer for every question anyone will ask. So, good answers are slow, costly, and never the only right one.

And training scores against one of them. That is the wall. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is so much easier about judging answers? Let's try to understand. In a simple way. You cannot cook like a chef.

But taste two dishes, and you know which is better. Everyone can judge dinner. Almost nobody can cook it. Answers are the same.

Writing a good one from a blank page is hard. Put two side by side, and picking the better one is easy. Watch the cost flip. One careful answer takes a skilled expert twenty minutes.

Choosing between two takes a reader thirty seconds. So flip who writes. The model produces both answers. The person just points at one.

The typing moves to the machine. Here is the real mechanism. The label used to be a whole answer, written by hand. Now the label is a pick.

A beats B. One bit. And the target problem dissolves. Nothing is matched word by word against one string.

Different wording is not wrong now. It only has to win. Why picks, and not a score out of ten? Because my seven is your five.

Numbers drift between people. A pick holds steady. It also lifts the ceiling. Copying stops at the writer you copied.

But you can spot the better answer without being able to write either. And it scales. Any question the model can answer, it can answer twice. Comparisons pile up as fast as the model types.

So, judging is cheaper, steadier, and not capped by the judge. The model writes, a person picks. That pile of picks teaches next. Quick check now.

One question is coming up. Let's see if it clicked.

How does training a judge with human preferences work? Let's try to understand. In a simple way. Start with what the picking gave us.

One question, two answers, a person pointing at the better one. Thousands of times over. But those people cannot stay. Training needs feedback millions of times, long after everyone went home.

No room of readers fits in that loop. So we train a second model to imitate the pickers. Its real name is a reward model. Call it the judge.

Its training data is that pile. Every row holds three things. The question, the answer people chose, and the one they turned down. Here is the real mechanism.

The judge is not built from scratch. It starts as a copy of the language model itself. Reading the answer is most of the job. Then you train that copy on one task.

See both answers, say which one the human chose. Guess wrong, nudge the weights. But we already know those picks. Why learn them?

Because the judge must rate answers nobody ever labeled. Generalizing is the whole point. So you test it. Hide some pairs, and see if it picks the same winner.

Good judges agree about seven times in ten. And the judge copies whoever taught it. If your readers quietly preferred longer answers, the judge now rewards length. Their habits are baked in.

So, a judge is a copy of the model, retrained on chosen and rejected pairs, to guess what a person would pick. Human taste, bottled. Quick check now. One question is coming up.

Let's see if it clicked.

How does a model score a response's quality? Let's try to understand. In a simple way. The judge does not write a review.

It reads, and it puts out a single number. One answer in, one number out. That number is the score. Inside, it is still the language model.

Same stack of layers, reading the words the same way. Only the very last part is different. That last part is called the head. In the language model, it turns the final layer into fifty thousand numbers.

One score per possible next word. The judge throws that head away, and bolts on a new one. It puts out exactly one number, instead of fifty thousand. That is the whole surgery.

What goes in matters too. The question and the answer are glued into one long document. Both, together. A brilliant answer to a different question is a bad answer.

Then it reads all the way to the final word. The score is taken at that last position, because only there has the model seen the whole thing. And the number has no units. It is not a percentage, and not out of ten.

It might be two point three. It might be minus one. So one score alone tells you nothing. It only means something beside another score, for the same question.

Higher wins. That is all it claims. So, the judge is the model with its word head swapped for a one number head. Question and answer in, one comparable score out.

Quick check now. One question is coming up. Let's see if it clicked.

How does turning human rankings into a reward signal work? Let's try to understand. In a simple way. A person looked at two answers and pointed at one.

That is all you have. An order. Nobody ever wrote down a score. But training runs on numbers.

To nudge weights, you need a loss to shrink. An arrow pointing at the better answer is not a loss. So the judge reads each answer alone, and puts out one number. Feed it the winner, feed it the loser.

Two numbers come back. Here is the real mechanism. Subtract them. Winner minus loser.

That one difference is the gap, and the gap is the training signal. Then squash the gap into a probability, between zero and one. A wide positive gap lands near one. The judge agrees with the person.

The loss is simply, make that probability higher. Ranked backwards? The gap goes negative, the probability drops, and the loss is large. So every pair pulls two ways at once.

The chosen answer's score is pushed up. The rejected answer's score is pushed down. Notice what never appears. A correct score.

Add ten to every number the judge gives, and every gap is unchanged. The loss cannot see it. And a longer ranking is just more pairs. Rank four answers, best to worst, and every couple becomes a row.

Six of them. So, rankings become a signal by scoring both, subtracting, and widening the gap. The number itself floats. Only the distance carries meaning.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the game a model plays against itself? Let's try to understand. In a simple way. Up to now, learning needed an answer a person wrote.

But the judge turns any answer into a number. So nobody writes one. So put two copies in the room. The player, the one being trained.

And the judge, which never changes. Both grew from the same model. Here is one move. Pull a question off a pile of prompts.

The player answers it, using the weights it has this second. That is a rollout. And it does not write its one best answer. It samples.

Same question, four attempts, four different answers. Variety is the point. The judge reads each attempt and hands back its number. Four attempts, four scores.

No person sits anywhere in that loop. Look at what appeared. Training data. The question came from the pile.

Every answer came from the model. It wrote its own homework. So who is it playing? Its own other attempts.

A score alone says nothing. It only matters beside what the same model just wrote. Here is the real mechanism. The weights shift, so the player writes different answers.

Every round is answered by the model the last round built. That is on policy. One rule keeps it honest. Only the player learns.

The judge stays frozen, so the bar never moves while the player climbs. So, the model answers, its own frozen copy scores, and the model that just learned answers again. Its own output is the curriculum. Quick check now.

One question is coming up. Let's see if it clicked.

How does updating a model with rewards work? Let's try to understand. In a simple way. The judge read the answer and handed back a number.

Seven out of ten. But a number alone changes nothing. No weight has moved. Earlier training had a target: a written answer to copy, word by word.

A reward gives no target. Just the model's own answer, and a score. So the rule flips. It says: the answer you just wrote, make it more likely.

Or less likely. Its own words become the target. Every word the model picked, it picked with some probability. Multiply those together and you get how likely the whole answer was.

That likelihood is the dial. Good score, turn it up. Bad score, turn it down. And the reward sets how hard.

But if every score is positive, every answer gets pushed up, even the bad ones. We need a bar. The model writes several answers to one question. Average their scores, and that average is the bar.

Above it, push up. Below it, push down. That gap has a name: the advantage. The score covers the whole answer, not each word.

So the same push lands on every word, the sharp ones and the filler alike. Each push stays small. One score is noisy. Leap on it and you wreck writing the model already had.

Tiny step, write again, score again. So rewards never hand the model an answer. They make its own better answers likelier, one small step at a time. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the balance between helpfulness and staying on topic? Let's try to understand. In a simple way. Training on rewards gives the model one instruction.

Score higher. Nothing else. Whatever the judge likes, do more of that. Follow that single instruction long enough and something strange happens.

The answers score better every round, and read worse. The model finds what the judge rewards. Length. Confident openings.

Endless polite hedging. It stuffs those in and stops answering the question. That works because the judge is only a stand in for a person. It learned from a few thousand comparisons.

Push past those, and it starts guessing. So we keep a second copy of the model, frozen at the moment reward training began. It never learns. It is only there to compare against.

For every answer, both models say how likely each word was. Compare those two, and you get how far the model has wandered. Now the score has two parts. The reward, minus the drift.

Being helpful earns points. Wandering off spends them. That subtraction is a leash. The model can still move.

It just pays for every step away from the way it used to talk. One knob sets how tight the leash is. Pull it too tight and nothing improves. Leave it slack and the model wanders into nonsense that scores well.

So, the reward pulls the model toward helpful. The frozen copy pulls it back toward normal. Good behavior lives where those two pulls settle. Quick check now.

One question is coming up. Let's see if it clicked.

How does skipping the reward model work? Let's try to understand. In a simple way. Look at what reward training needs.

Two trainings. The judge first, then the model. Three big models in memory. All to teach one preference.

Where did that judge come from? A pile of chosen and rejected answers. We squeezed the pile into one scorer, and the pile went in a drawer. Squeezing loses things.

The judge guesses on answers it never saw. And the model chases those guesses, because the judge is all it hears. Which raises an awkward question. The real preferences are still in that drawer.

Why learn them second hand, from a stand in? So DPO does the obvious thing. Direct Preference Optimization. Throw the judge away.

Feed the chosen and rejected answers straight into the model being trained. And the shape goes back to something familiar. One fixed dataset. A few passes over it.

No sampling, no scoring, no loop. But it is not copying. Fine tuning says, write this. DPO says, prefer this one over that one.

The rejected answer teaches too. One piece survives. The frozen copy stays, holding the model near where it started. DPO builds that leash in, instead of bolting it on.

The cost is freshness. A judge could score any new answer the model invented. DPO only ever sees the pairs you already collected. So, the reward model was a middleman.

DPO deletes it and learns from the preferences directly. Less machinery, and one less thing to game. Quick check now. One question is coming up.

Let's see if it clicked.

How does learning straight from preferences work? Let's try to understand. In a simple way. Here is what DPO gets.

One question, and two answers. A person marked one chosen, the other rejected. That is all. No score anywhere.

But the old rule needed a number. Turn this answer up by seven. Here there is no seven. Just an order.

This one beats that one. So we ask the model itself. How likely was the chosen answer? How likely was the rejected one?

Two numbers, and the model already knows them both. Now the preference is easy to say. The chosen answer should be the likelier one. So push its likelihood up, and push the rejected one down.

Same step, both moves. But raw likelihood lies. Bland, common answers look likely to any model. So compare each one against the frozen copy from before training.

Not how likely. How much likelier. And that ratio is the trick. It behaves exactly like a reward.

DPO never deleted the judge. It hid the judge inside the model being trained. The push is soft. Pairs the model already ranks right barely move it.

Pairs it ranks backwards move it hard. The effort lands where it is still wrong. One knob sets the strength. Beta.

Turn it up and the model stays close to the frozen copy. Turn it down and it chases the preferences further. So, DPO reads a pair, measures both answers against the frozen copy, and widens the gap. The preference is the whole training signal.

Quick check now. One question is coming up. Let's see if it clicked.

How does learning from watching videos work? Let's try to understand. In a simple way. Writing answers is slow.

Ranking them is slow too. But people already record themselves doing useful work. Millions of hours. Nobody labeled a frame.

And a demonstration is an answer. Someone booking a flight, fixing a file. Nobody typed it out. They performed it, and the recorder kept it.

But a video is only pixels. Frame after frame of what the screen looked like. It never shows the clicks or the keys. And training needs a pair.

Something in, something to copy. The screen goes in. The thing to copy, the action, was never saved. So rebuild it.

Take two frames in a row and ask what action explains the change. Cursor slid right? A mouse move. That guesser is an inverse dynamics model.

It needs real examples first. So you pay a small group to work while every key and click is logged. Thousands of hours, not millions. Why is that job easier?

The guesser sees the frame after. It only says what happened. It never decides what was smart. Now unleash it.

Run it across the whole unlabeled pile and stamp an action onto every frame. The mountain just grew labels. And now it is ordinary training again. The frame is the question, the stamped action is the answer.

The model copies it. Behavior cloning. One limit. It copies whoever it watched, mistakes included.

So watching builds a competent starting model. Preferences and rewards push it further. Quick check now. One question is coming up.

Let's see if it clicked.