Frontier models, torn down

What exactly is the shared blueprint of frontier models? Let's try to understand. In a simple way. Different labs, different names.

GPT, Claude, Llama, Gemini. They act like fierce rivals. But if you open the hood, you find almost the same machine inside. That machine is called the Transformer.

Nearly every frontier model is built from this one core design, first published back in twenty seventeen. It always begins the same way. Your text is broken into tokens, and each token turns into a list of numbers the model can handle. The heart of the design is a single repeating unit.

We call it a Transformer block. Inside it, two parts do all the work. The first part is attention. Every token looks at the others and decides which ones matter for what comes next.

The second part is a feed forward layer. It reshapes each token on its own, mixing in patterns the model soaked up during training. Now the trick. That same block is copied and stacked, dozens of times.

Each copy has the same shape, just its own weights. To keep the signal alive through so many layers, each block adds its result back onto its input. That shortcut is called a residual. So what really separates these models?

Mostly their size and their training, not their shape. The recipe is shared. The scale is what differs. So the blueprint is simple.

Tokens in, a tall stack of identical blocks, one next word out. Quick check now. One question is coming up. Let's see if it clicked.

What exactly are a model's billions of parameters? Let's try to understand. In a simple way. A parameter is just one number.

A single value the model learned and stored inside itself. Not something you type in. It is a weight it set for itself during training. Picture each parameter as a tiny dial.

Turn them all to the right settings, and together they hold everything the model knows. Its grammar, its facts, its style. Where do they live? Inside every Transformer block, in big grids of numbers called weight matrices.

Attention holds some. The feed forward part holds far more. To count them, take each grid and multiply its width by its height. Then add up every grid, across every layer.

That is the parameter count. Add it all up and the number is staggering. A small open model holds a few billion. The biggest reach hundreds of billions, and beyond.

This is what the B in a name means. Seven B, seventy B, four zero five B. That letter is just billions of parameters, the model's rough size. Why keep adding more?

More parameters give the model more room to store patterns. But each one costs memory to hold, and compute to run. But bigger is not automatically smarter. A well trained smaller model can beat a bloated larger one.

The count sets the ceiling, not the score. So a parameter is one learned number. Count every grid in every layer, and you land in the billions. That count is the model's size, not its smarts.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the context a model reads before it answers? Let's try to understand. In a simple way. Here is the strange part.

A frontier model has no memory of its own. The moment before it answers, its mind is blank. So before it writes a single word, it reads one big bundle of text. That bundle is called the context, or the prompt.

The bundle has layers. First, a hidden system prompt. Instructions the app sets before you ever type, telling the model who it is and how to behave. Next, the whole conversation so far.

Every earlier message, yours and the model's own replies, pasted right back in. Then, at the very end, your newest message. The actual question you just asked, sitting last in line. Often there is more.

Attached files, searched documents, tool results. All of it stitched into the same stream of text. And here is the twist. The model keeps nothing between turns.

Every reply, the app resends this entire bundle from scratch. But the bundle cannot grow forever. It has to fit a fixed budget called the context window, measured in tokens. Overflow, and the oldest text drops off.

So the model only knows what sits in that window right now. Nothing outside it exists. That is why a clear, complete prompt matters so much. So before it talks, a model reads its whole context.

System prompt, history, and your message. Only then does it guess the next word. Quick check now. One question is coming up.

Let's see if it clicked.

How does filtering the internet for training data work? Let's try to understand. In a simple way. A model learns from text scraped off the open web.

But the raw web is a giant mess. Billions of pages of spam, menus, broken code, and pure gibberish. Why bother cleaning it? Because a model copies the patterns in its data.

Feed it junk, and it learns to write junk. Garbage in, garbage out. So the first step is to remove duplicates. The same paragraph can appear thousands of times across the web.

Keep one copy, and throw the rest away. Next, a quality classifier scores every page. A small model asks, does this read like a careful article or a book? High scores stay.

Low scores are dropped. Cheap rules catch the obvious trash. Too few words, too many symbols, or a known spam domain. These simple filters toss whole pages in an instant.

Some text is filtered for a different reason. Toxic pages and private details are stripped out. Test questions are removed too, so the model cannot simply memorize the answers. What comes out is far smaller than what went in.

Trillions of raw words become a leaner, cleaner pile. And for training, quality beats raw quantity almost every time. So filtering is like a funnel. Crawl the whole web, strip away the junk, and keep the clean writing that is actually worth learning from.

Quick check now. One question is coming up. Let's see if it clicked.

How does learning to predict the next word work? Let's try to understand. In a simple way. The whole training of a language model is one simple game.

Show it some text, and ask it to guess the word that comes next. Take a sentence. The cat sat on the mat. Cover the last word, and the model must guess it.

That is the whole task. Where does the right answer come from? From the text itself. The next word is already sitting right there.

So every sentence is a free practice question, answer included. The model does not blurt out one word. It spreads its bet across the whole vocabulary. Mat gets a high chance, banana almost none.

A full probability distribution. Then it peeks at the real next word. Confident and right, barely a nudge. Wrong, a bigger correction flows back and shifts the weights.

Now repeat this over trillions of words. Each nudge on its own is tiny. But together they slowly tune billions of weights toward better guesses. Here is the surprise.

To guess the next word well, the model must soak up grammar, facts, and a little reasoning. One simple game teaches all of it. And this is also how it writes for you. Predict a word, stick it onto the end, then predict again from the longer text.

One word at a time. So the whole thing rests on one move. Guess the next word, check the answer, adjust. No human labels needed.

The text teaches the model itself. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is instruction tuning? Let's try to understand. In a simple way. A freshly trained model is really just an autocomplete engine.

It read piles of text and learned one trick. Predict the next word. So if you ask it a question, it may not answer. It might just add more questions, or ramble on.

On the open web, a question is usually followed by more text, not a helpful reply. The fix is a second, shorter round of training. It is called instruction tuning, or supervised fine tuning. Now humans write thousands of example pairs.

Each pair is an instruction, and the ideal response a good assistant would give. The model studies these pairs the same old way. Predict the next word. But now the words it copies are polished answers, so it learns to imitate them.

Nothing new gets bolted on. The very same weights are simply nudged, a little, toward responding instead of just continuing the text. It also learns the shape of a conversation. Where your turn ends, where its turn begins, and when to stop talking.

One catch. This step does not pour in new facts. Those came earlier. Tuning only shapes behavior.

Be helpful, follow the instruction, stay on task. So instruction tuning takes a raw text predictor, and with a small set of good examples, turns it into an assistant that does what you ask. Quick check now. One question is coming up.

Let's see if it clicked.

How does teaching AI with human preference scores work? Let's try to understand. In a simple way. After a model learns to follow instructions, it can answer in many valid ways.

For an open question there is no single correct answer to check against. So predicting the next word cannot tell which reply is better. So instead of grading, we ask a person to compare. Show one prompt and two responses from the model, side by side.

Call them A and B. The person just picks the one they prefer. Not a grade from one to ten, simply which answer is better. Choosing between two is faster and more consistent than scoring each alone.

Do this thousands of times, with many people and many prompts. Now you have a big pile of preference data. Each item holds a prompt, two answers, and the winner. Those human choices turn into a score.

The preferred answer earns a high number, the rejected one a low number. That score becomes the reward the model chases. Now training begins. Response by response, the model is nudged toward answers with a higher preference score, and away from ones people rejected.

This loop is reinforcement learning from human feedback. What changes? The model stops merely predicting likely text and starts giving what people actually want. More helpful, more honest, less rambling.

That shift is what we call alignment. So, human preference scores teach the model taste. People compare answers, the winners become a reward, and the model learns to give replies we prefer. Quick check now.

One question is coming up. Let's see if it clicked.

How does training a separate AI to judge responses work? Let's try to understand. In a simple way. To keep improving, the model tries answer after answer, each attempt needing a score.

But a person cannot rate millions. Humans are too slow for that loop. So we train a second, separate AI whose only job is to judge. It is called a reward model, an automatic critic that stands in for the human.

Where does its taste come from? The pile of human comparisons collected earlier: a prompt, two answers, and the winner a person picked. The reward model reads a prompt and one response, and outputs a single number. A quality score, higher when people like it more.

Training it is simple. Show it a winning answer and a losing answer, and adjust its weights until it scores the winner higher. Repeat across every pair. The judge is usually a copy of the language model, its word guessing head swapped for a single score.

It already knows language, it just learns to rate. Now the human steps out of the loop. The judge scores millions of responses instantly, day and night. That is what makes large scale training possible.

One catch. The judge is only as good as the people it copied. And the model can learn to game it, chasing answers that fool the judge, not better ones. So, a reward model is a separate AI, trained on human comparisons, that scores any response.

It stands in for the human and scales training up. Quick check now. One question is coming up. Let's see if it clicked.

How does Claude learning from a constitution work? Let's try to understand. In a simple way. Teaching a model to be safe once leaned on people.

For each answer, a human had to judge if it was harmful, one by one. Slow, costly, and the values lived only inside their heads. The new idea is simple. Write the values down.

A constitution is a short list of plain rules. Be helpful. Be honest. Avoid harm.

Then the model grades its own work. It writes a first answer, picks one rule from the constitution, and criticizes its reply against that rule. Next it rewrites the answer to follow that rule better. Critique, revise, repeat across the rules.

Then the model is fine tuned on these improved answers. Now the second phase. The model makes two answers to a prompt. Another AI compares them and picks the one that follows the constitution better.

Those choices become the reward. The model is trained toward the answers the rules prefer. This is reinforcement learning from AI feedback. Why do this?

The values are now written down. You can read them, edit them, and apply them consistently, without an army of human labelers. One thing stays human. People still write the constitution.

The rules carry the human values, and the AI just applies them at a scale no team of raters could reach. So, a constitution is a written list of principles. The model critiques and revises against it, and an AI judge scores by it, replacing most human feedback. Quick check now.

One question is coming up. Let's see if it clicked.

How does using expert sub models for faster thinking work? Let's try to understand. In a simple way. A normal model is called dense.

For every single word you send it, the whole network runs. Every parameter does math, even the parts that have nothing useful to add. Powerful, but slow and expensive. So here is a smarter idea.

Take that one big block and split it into many smaller sub networks. Each one is called an expert. You might have eight of them, or dozens. Now, in front of the experts sits a tiny helper called the router.

It reads each word and decides which experts should handle it. And here is the trick. The router does not wake all of them. It picks just the top two, and the rest stay asleep for that word.

So even if the model holds a huge number of experts, only a small slice actually runs for each word. Engineers call this sparse activation. Big model, small bill. Over training, the experts drift apart.

One leans toward code, another toward grammar, another toward numbers. The router slowly learns who to ask for each kind of word. The payoff is clear. You get the knowledge of a giant model, but you pay for only the few experts that woke up.

That is why mixture of experts models think so fast. So, split the network into experts, let a router pick a few per word, and skip the rest. Huge capacity, tiny cost, faster thinking. Quick check now.

One question is coming up. Let's see if it clicked.

How do models learn to see and listen? Let's try to understand. In a simple way. A language model reads only tokens, numbers standing for text.

A photo or a sound clip is not text, so the model cannot see or hear it. So here is the trick. We turn every input into the same numbers the model already reads. Little lists called vectors.

Take an image. We cut it into a grid of small patches, and a vision encoder turns each patch into one vector. Sound works the same way. We slice the clip into tiny time windows, and an audio encoder turns each slice into a vector.

But there is a catch. A projection layer must nudge these vectors into the same space as text, so a dog photo sits beside the word dog. Now everything is the same shape. The model sees one long line of vectors, text and image and sound mixed, and runs the same next word machine over it.

How do the encoders learn this? We train them on pairs, pictures beside captions, audio beside transcripts, until their vectors line up with the matching words. That is why you can hand it a photo, a voice clip, and a question at once. It reasons across all three.

One model, many senses. So, encoders turn images and sound into vectors, a projection drops them beside the word tokens, and one transformer reads the whole mix. Quick check now. One question is coming up.

Let's see if it clicked.

How does predicting model skill before you train it work? Let's try to understand. In a simple way. Training a frontier model costs tens of millions of dollars and takes months.

Imagine building the whole thing, then finding out it came out weak. The number a model tries to lower is its loss. Roughly, how wrong its guesses are. Here is the surprise.

Scale a model up, and that loss falls in a smooth, predictable way. That fall is driven by three knobs. More parameters, the model's size. More training data.

And more compute. Push all three, and the loss keeps sliding down. Even better, the shape is known. Plot the loss on a log chart, and the points fall along a straight line.

This is called a power law. Easy to extend. So here is the recipe. Train a handful of small, cheap models at different sizes.

Each takes hours, not months. Then mark where each one lands, and draw the line through them. Now extend that line far to the right, out to the giant model you have not built yet. Where it lands is your prediction.

You know the loss before spending a dollar. This really happened. Before training GPT four, OpenAI predicted its score from models a thousand times smaller. The real model landed right on the line.

Scaling laws do one more thing. They tell you the right balance of data and size. But one caution. The curve predicts loss, not every surprising skill that can emerge.

So, skill scales predictably. Train tiny models, fit a straight line, extend it, and read off how good the giant will be. Before you pay for it. Quick check now.

One question is coming up. Let's see if it clicked.

How does a model's long memory work? Let's try to understand. In a simple way. First, what is memory here?

A model keeps no notebook. Its memory is simply all the text it can see at once: the context window. How does it hold all of it at once? A trick called attention.

Every word compares itself with every other word, and keeps what matters. But that is why long memory is hard. If every word checks every other word, doubling the text quadruples the work. The cost is the wall.

A second problem. The model must know where each word sits. Rotary position encoding counts distances between words, not fixed slots, so it can reach past its training length. To beat the cost, models get picky.

With sliding window attention, each word looks only at a nearby band plus a few anchors, not the whole history. One more saver. As the model writes, it stores a small summary of each past word: the key value cache. Now it never re-reads the whole history.

Together, these tricks paid off. Windows jumped from a couple thousand words to hundreds of thousands. Some models now hold over a million. But long is not perfect.

A fact buried in the middle can slip past, while the start and end stand out. Where you place things still matters. So, long memory is attention over a window, stretched by smart positions, trimmed by sliding attention, and sped up by a cache. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is Llama's openness? Let's try to understand. In a simple way. Most frontier models are closed.

You send text to a server, an answer comes back, and that is all you get. The trained weights stay locked away. Llama is different. Meta lets anyone download the finished weights, the billions of numbers the model learned.

This is called an open weight model. But here is the catch. Open weights is not open source. You get the trained result, not the training data, and not the exact recipe behind it.

So what can you do with the weights? You can run the model on your own machine. No API bill, and your data never leaves the room. You can also fine tune it.

Keep training those open weights on your own examples, shaping the model for your task. A closed model never lets you touch them. And here is the first reveal. Peek inside, and the architecture is the same Transformer everyone else uses.

No secret shape hides in the design. So if the blueprint is public, where is the edge? It lives in the data, the compute, and the training recipe. That is the real secret.

One more payoff. Because the weights are open, researchers can inspect them and build on them. A closed model can only be poked from outside. So, open weights means you can download and run Llama yourself.

The design is shared. The real moat is the data and the training behind it. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is the GPU cost of training GPT four? Let's try to understand. In a simple way. Training GPT four cost more than one hundred million dollars.

Not salaries, not offices. Just chips doing math, for months. Where does the money go? Almost all of it is compute.

Racks of graphics chips, called GPUs, doing arithmetic at a huge scale. How huge? Training GPT four took about two, followed by twenty five zeros, math operations. More than the stars in the observable universe.

The problem is, even the fastest single chip, running non stop, would need hundreds of years to finish it alone. So the work is split across a cluster. Around twenty five thousand GPUs, wired together, crunching pieces of the same model at once. And chips are not cheap.

Renting that many for three months, at a couple of dollars an hour, is where the tens of millions go. They also drink power. The cluster pulls tens of megawatts, enough to light a small town. The electricity bill alone runs into the millions.

It gets worse. Over months, chips crash and runs blow up. Every restart burns more GPU time, and you pay for all of it. This is why only a few labs can build a model this big.

One run can cost more than a whole company is worth. So, GPT four is costly because compute is costly. Thousands of GPUs, months of power, and failed runs. Just to train it once.

Quick check now. One question is coming up. Let's see if it clicked.