What exactly is pretraining? Let's try to understand. In a simple way. A model starts out knowing nothing.
Its numbers are random. Pretraining is the first stage, where the model reads a huge slice of the internet and picks up language. And here is the surprising part. There is only one exercise.
Predict the next token. Show the model some text, hide what comes next, ask it to guess. Take the sentence, the cat sat on the. The model guesses mat.
The real text says mat too. Great. Guessed wrong? Its numbers get nudged a little.
Notice something. Nobody labeled anything. The next word is already sitting right there in the text. The answer comes for free.
Researchers call this self-supervised learning. The data supervises itself. That is what makes reading trillions of tokens possible. No human could write that many answer keys.
Now, why does one dumb game teach so much? Because predicting well is hard. To guess the next word in a physics article, you need some physics. Grammar, facts, even reasoning come along free.
And this happens at absurd scale. Trillions of tokens, billions of tiny nudges. Together they turn random numbers into a model of language. The result is called a base model.
Not a chatbot yet. Ask it a question, and it may just continue with more questions. It completes text, it does not converse. So, pretraining is one game played on the internet.
Guess the next token, get corrected, repeat. Everything else builds on top of this. Quick check now. One question is coming up.
Let's see if it clicked.
What exactly is training data? Let's try to understand. In a simple way. A language model learns by reading text.
Trillions of words of it. And someone has to actually collect all that text. So where do you find that much writing? The biggest source is the open web.
Projects like Common Crawl visit billions of pages and save a copy of the text. That giant pile is the raw material. But here is the thing. Most of the raw web is junk.
Spam, ads, menus, auto generated filler. Train on junk, and the model learns junk. So teams filter aggressively. Quality classifiers score every page, keep the ones that read like real writing, and drop the rest.
Often ninety percent of the crawl gets thrown away. Next comes deduplication. The same article gets copied across thousands of sites. Seeing it once is learning.
Seeing it a thousand times is just memorizing. Then the premium sources. Books, Wikipedia, academic papers, and code from public repositories. These piles are smaller, but far cleaner.
Now the key decision: the data mixture. Teams set a recipe. So much web, so much code, so many books. Clean sources get upweighted, so the model sees them more often.
One more thing. The supply of good public text is running low. So labs now add synthetic data, where a strong model writes fresh training text for the next one. So, training data is mostly the open web, filtered hard and deduplicated, then mixed with books, papers, and code in a careful recipe.
Quick check now. One question is coming up. Let's see if it clicked.
What exactly is loss? Let's try to understand. In a simple way. During training, the model keeps guessing the next token.
Loss is one number that says how wrong that guess was. Big number, bad guess. Small number, good guess. Here is the thing.
The model never guesses just one token. It spreads probability across its whole vocabulary. Like placing bets on every possible next word. Then we peek at the real next token in the text.
One question. How much probability did the model put on that correct token? Say the text reads, the cat sat on the. The right token is mat.
If the model gave mat ninety percent, loss is tiny. If it gave mat one percent, loss is huge. The standard version is called cross entropy loss. Take the probability on the correct token.
Take the log. Flip the sign. That's it. Why the log?
Because it punishes confident mistakes brutally. Dropping from ten percent to one percent hurts far more than dropping from ninety to eighty. One guess is noisy. So training averages the loss over huge batches of text.
Millions of predictions collapse into one honest score. And that score is the whole point. Training is just one instruction, repeated forever. Adjust the weights so the loss goes down.
How that adjustment works is the next lesson. So, loss is a single number scoring how surprised the model was by the real text. Lower loss, better predictions. It is the only feedback the model ever gets.
Quick check now. One question is coming up. Let's see if it clicked.
How does gradient descent work? Let's try to understand. In a simple way. A model is a huge pile of numbers called weights.
Billions of tiny knobs. Training means finding the knob settings that make the loss small. Picture loss as a landscape. Each knob setting is a spot, its height is the loss.
We want the lowest valley. But it's foggy. You only feel the slope under your feet. That slope has a name.
The gradient. For each weight, it says which way the loss goes if you nudge that weight up. So you step the opposite way. Downhill.
Try it with one knob. Nudge it up a little. The loss rises. So turn it down instead.
Now imagine that for billions of knobs, all at once. Here is the clever part. You never test knobs one by one. Backpropagation uses the chain rule from calculus to compute every gradient in one backward pass.
Next, how big a step? That is the learning rate. Too big, and you leap over the valley and bounce around. Too small, and training crawls.
One more trick. Checking all the data every step is too slow. So we use a small random batch each time. Noisy, but cheap.
That is stochastic gradient descent. Then you repeat. Forward pass, measure the loss, backward pass, small step downhill. Millions of times.
The weights drift toward a setting that works. So, gradient descent is the engine of training. Feel the slope, take a tiny step downhill, repeat until the loss stops falling. Quick check now.
One question is coming up. Let's see if it clicked.
How does the cost of training work? Let's try to understand. In a simple way. Here is the thing.
Training is just arithmetic. For every token the model reads, it runs a huge pile of multiplications across all of its weights. There's a standard rule. Each token costs about six operations per parameter.
So total compute is six times parameters times tokens. That's the whole bill. Now plug in real numbers. Seventy billion parameters.
Fifteen trillion tokens. Multiply it out, and you get a number with about twenty-five digits. Operations, not dollars. Yet.
One GPU, even a top one, would grind on that for centuries. So labs run thousands of GPUs in parallel, around the clock, for months. And GPUs are not cheap. One high end chip rents for a few dollars per hour.
Multiply by ten thousand chips, by months of hours, and you're past twenty million. There's also the power bill. A big training cluster draws tens of megawatts. That's the electricity of a small town, plus the cooling to keep it alive.
Here is the hidden cost. With thousands of chips, something fails every few hours. So runs save checkpoints and restart from the last one. Lost time is money.
And GPUs rarely run flat out. They wait on data moving between chips. Real utilization often sits near forty percent. You pay for idle compute too.
So, the bill is simple math. Six times parameters times tokens, turned into GPU hours, plus power, plus failures. Small model, small bill. Frontier model, millions.
Quick check now. One question is coming up. Let's see if it clicked.
What exactly is instruction tuning? Let's try to understand. In a simple way. Here is the thing.
A freshly pretrained model is not an assistant. It just continues text. Ask it a question, and it might reply with three more questions. Why?
Because on the internet, questions are often followed by more questions. The model learned to predict what comes next, not to be helpful. Instruction tuning fixes that. Humans write example conversations.
An instruction, and the ideal response. Things like, summarize this email, then a clean summary. Then we keep training the model on those pairs. Same model, same learning process, just new data that shows how an assistant behaves.
The literature calls this supervised fine tuning, or SFT. Supervised, because a human supplied the target answer for every single prompt. Under the hood, nothing exotic. It is still next token prediction.
But the loss is usually computed only on the response tokens, so the model learns to answer, not to imitate the question. And the dataset is tiny. Pretraining eats trillions of tokens. Instruction tuning may use just tens of thousands of curated examples.
Quality beats quantity here. So it does not add much new knowledge. The knowledge came from pretraining. Instruction tuning unlocks it, by teaching the format, the tone, the habit of following orders.
So, instruction tuning turns a text predictor into an assistant. Show it good demonstrations, train on the answers, and it starts to obey. Quick check now. One question is coming up.
Let's see if it clicked.
What exactly is RLHF? Let's try to understand. In a simple way. After instruction tuning, a model can follow orders.
But some answers are helpful, some are lazy, some are subtly wrong. And here is the thing. Nobody can write a loss formula for a good answer. But humans can compare.
Show a person two answers to the same prompt, and they will point at the better one. That single choice is called a preference. So labelers do this at scale. One prompt, two answers, pick the better one.
Repeat hundreds of thousands of times. Now you have a giant pile of human rankings. Next, train a second model on that pile. The reward model.
It reads any answer and spits out one number. A score for how much a human would like it. Now the reinforcement learning part. The chat model writes an answer, the reward model scores it, and the weights get nudged toward higher scores.
The classic algorithm here is PPO. One crucial leash though. A KL penalty keeps the model close to where it started. Chase the score with no leash, and the model drifts into weird text that only the judge loves.
That failure has a name. Reward hacking. The reward model is just a proxy for people, so the model learns to please the judge. Long, confident, flattering answers can score high while being wrong.
So, RLHF in one line. Humans rank answers, a reward model learns their taste, and reinforcement learning tilts the chat model toward it. Quick check now. One question is coming up.
Let's see if it clicked.
What exactly is a scaling law? Let's try to understand. In a simple way. Every model has three big dials.
How many parameters. How much training data. How much compute. Scaling laws describe what happens when you turn them up.
Here is the surprise. Scale up, and the loss falls along a smooth, predictable curve. On a log plot, it is nearly a straight line. That shape has a name.
A power law. Double the model, and loss drops by a fixed, repeatable fraction. It kept holding over many orders of magnitude. And that is the superpower.
You can train small models, fit the curve, and forecast how good a huge model will be. Before you ever build it. In twenty twenty, Kaplan and team at OpenAI measured these laws. Their advice: size matters most.
Make the model big, even if data stays modest. Then DeepMind's Chinchilla paper flipped it. For a fixed compute budget, most models were too big and undertrained. The sweet spot: roughly twenty tokens of data per parameter.
So the real rule is balance. Grow parameters and data together. A giant model starved of data wastes compute. So does a tiny model drowning in it.
One more twist. Loss falls smoothly, but abilities can jump. Arithmetic, translation, reasoning. They seem to switch on once a model crosses some scale.
So, scaling laws say bigger wins, predictably. Loss follows a power law in parameters, data, and compute. And the winning move is scaling all three in balance. Quick check now.
One question is coming up. Let's see if it clicked.