What exactly is the problem the transformer solved? Let's try to understand. In a simple way. Before twenty seventeen, language models ran on recurrent networks.
RNNs, and their upgrade, LSTMs. They read a sentence one word at a time, left to right. Each step squeezed everything read so far into one fixed-size memory. So by word fifty, word one was a blurry summary.
Long sentences just faded. Try this. The keys to the old cabinet in the hallway are missing. To say are, not is, you must hold keys across that whole gap.
RNNs kept fumbling it. Now the second problem. Step twenty cannot start until step nineteen finishes. Everything is sequential.
GPUs, built to do thousands of things at once, mostly sat waiting. Training made it worse. The learning signal flows backwards through every step, shrinking as it goes. The vanishing gradient problem.
Distant words barely teach the model anything. Then in twenty seventeen, a Google team published Attention Is All You Need. The transformer. The move was radical.
Throw out recurrence completely. In its place, a mechanism called attention gives every word a direct connection to every other word. Word one to word five hundred? One hop.
No relay, no fading. And since no word waits on the previous one, the whole sentence is processed at once. Training finally fills the GPU. That made huge models practical.
So, the transformer solved two problems. Memory fading over long distances, and one word at a time slowness. Direct connections, all in parallel. Quick check now.
One question is coming up. Let's see if it clicked.
How does self-attention work? Let's try to understand. In a simple way. Take the word bank.
River bank, or money bank? The word alone can't tell you. Meaning depends on the neighbors. So the model needs a way to mix that context in.
Here is the trick. Every word looks at every other word in the sentence. And it asks a simple question. How relevant are you to me, right now?
To do that, each word makes three vectors. A query, what am I looking for. A key, what do I offer. And a value, the actual information I carry.
Now the matching. My query meets your key, and a dot product scores the pair. Big score, you matter to me. Small score, not so much.
Those raw scores go through a softmax. It squashes them into weights that add up to one. Like splitting one hundred percent of attention across the sentence. Then each word takes a weighted sum of everyone's values.
Heavy weight, big contribution. The word ends up with a new vector, a blend of the relevant context. Concrete example. The animal didn't cross the street because it was tired.
Who is it? Self-attention puts a heavy weight from it onto animal. The meaning snaps into place. And here is the thing.
Nobody hand-writes these rules. The maps that make queries, keys, and values are learned matrices, tuned during training. So, self-attention. Every word scores every other word, turns the scores into weights, and blends the values.
Context, computed on the fly. Quick check now. One question is coming up. Let's see if it clicked.
How does multi-head attention work? Let's try to understand. In a simple way. Self-attention lets each word look at the others and pull in what it needs.
But one attention pass asks just one question. One way of relating words. Here is the thing. Language runs on many relationships at once.
Who did what. What does this pronoun point to. Which words sit nearby. So the transformer runs several attentions in parallel.
Each one is called a head. Like a committee, where every member reads the same sentence with a different focus. Take, the cat chased the mouse because it was hungry. One head links it back to cat.
Another links chased to mouse. A third watches neighboring words. How? Each head gets its own learned projections.
Its own queries, keys, and values. So each head scores word pairs in its own way, and pulls out different information. Here is the clever part. The vector gets split.
Say seven hundred sixty eight numbers, twelve heads, sixty four each. So many heads cost about the same as one big one. Then the heads report back. Their outputs get concatenated into one long vector, and a final output projection mixes them.
One answer per word, built from many viewpoints. And in trained models, this really happens. Researchers find heads tracking syntax, heads following pronouns, heads staring at the previous token. Some turn out redundant, and can be pruned.
So, multi-head attention is many small attentions in parallel, each with its own projections, catching different relationships. Concatenate, mix, done. Quick check now. One question is coming up.
Let's see if it clicked.
How does positional encoding work? Let's try to understand. In a simple way. Here is a strange fact about self-attention.
On its own, it treats your sentence as a bag of tokens. No first, no last. Just a set. So, dog bites man, and man bites dog.
Same three tokens. Without position info, attention computes the exact same relationships for both. That's clearly broken. The fix is simple.
Before anything else, we stamp each token with where it sits. Position one, position two, and so on. Each position gets its own vector. And here is the neat part.
We simply add that position vector to the token's embedding. One sum, and every token now carries its meaning plus its address. The original transformer used sinusoidal positional encodings. Sine and cosine waves at many different frequencies.
Slow waves, fast waves, all sampled at each position. Why waves? Two reasons. Every position gets a unique fingerprint.
And shifting by a fixed offset becomes a simple, learnable pattern. So relative distance is easy to read. GPT took another route. Learned position embeddings.
A trainable table, one vector per position, tuned during training like any other weight. Most modern models use rotary position embeddings, or RoPE. Instead of adding, they rotate the query and key vectors by an angle that grows with position. Relative distance falls out naturally.
So, attention alone forgets order. We rescue it by injecting position, added waves, a learned table, or a rotation. That's how word order survives. Quick check now.
One question is coming up. Let's see if it clicked.
What exactly is the feed-forward layer? Let's try to understand. In a simple way. Attention gets all the fame.
It lets words look at each other. But once a word has gathered its context, something still has to think about it. That job belongs to the feed-forward layer. Here is the key rule.
The feed-forward layer works on each position separately. No peeking at neighbors. The same small network runs on every word, one by one. Inside, it's simple.
Two linear layers, with a nonlinearity in between, usually ReLU or GELU. Multiply, bend, multiply again. That's the whole recipe. Now the shape.
The first layer expands the word's vector, usually four times wider. So seven hundred sixty eight numbers become about three thousand. The second layer squeezes it back down. Why the wide middle?
Room to compute. Each of those thousands of hidden units can detect one pattern in the word's vector. The nonlinearity switches them on or off. And here is the surprising part.
Research suggests these layers act like key-value memories. Facts like, Paris goes with France, seem to be stored largely in the feed-forward weights. One more thing. Roughly two thirds of a transformer's parameters live in these layers.
Attention is the famous part. Feed-forward is the heavy part. So, attention gathers context between words. The feed-forward layer then processes each word alone.
Expand, bend, compress. That's where much of the model's knowledge sits. Quick check now. One question is coming up.
Let's see if it clicked.
What exactly is the plumbing inside a transformer? Let's try to understand. In a simple way. A transformer is a tall stack of identical blocks.
Attention, then feed-forward, repeated dozens of times. And here is the problem. Signals passing through that many layers tend to fade or blow up. The fix is a residual connection.
Each layer does not replace its input. It adds to it. The output is the input plus whatever the layer computed. A small update on top.
Think of editing a draft. Each layer scribbles one improvement in the margin. The original text is never thrown away. So even if one layer writes nothing useful, the draft survives.
This also rescues training. Gradients flow backwards through that addition completely untouched, like a highway. Without residuals, deep networks hit vanishing gradients, and layers near the bottom barely learn. People call this the residual stream.
One shared highway of vectors running through the whole model. Attention reads from it, writes back. Feed-forward reads from it, writes back. But keep adding updates and the numbers drift.
Some grow huge, some shrink. So layer norm rescales each token's vector to a steady mean and spread, at every layer. Placement matters too. Modern GPT style models use pre-norm.
They normalize before attention and before feed-forward, not after. That keeps training stable even at extreme depth. So the recipe for one block is simple. Normalize, apply attention, add it back.
Normalize, apply feed-forward, add it back. Boring plumbing, but it's why hundred layer models train at all. Quick check now. One question is coming up.
Let's see if it clicked.
What exactly is a decoder-only transformer? Let's try to understand. In a simple way. The original transformer from twenty seventeen had two halves.
An encoder that read the input, and a decoder that wrote the output. It was built for translation. Then GPT made a bold move. It threw the encoder away and kept only the decoder stack.
One tower of layers, doing one job. And that job is next token prediction. Read everything so far, then guess the single next piece. The whole model exists for that one guess.
Try it. The cat sat on the. Your brain says mat. The model does the same, but outputs a probability for every token in its vocabulary.
There is one strict rule. The causal mask. Each token can only look at tokens before it, never after. Otherwise guessing the next word would be cheating.
Generation is just a loop. Pick a token, stick it on the end, feed everything back in. Repeat. That's autoregressive decoding.
Every chat reply is written this way. Here is the beautiful part. Training needs no labels. Any text is its own answer key.
The next word is right there. That's self supervised learning, and it unlocks the whole internet as data. And that's why this design took over. GPT two, GPT four, Llama, Claude, Gemini.
Nearly every modern LLM is a decoder-only transformer. Same recipe, just bigger. So, decoder-only means one stack, one rule, one job. Look left only, predict the next token, and loop.
That's the engine inside today's giants. Quick check now. One question is coming up. Let's see if it clicked.
How does transformer scaling work? Let's try to understand. In a simple way. Here is the strange thing about transformers.
Make one bigger, feed it more data, and it just gets better. No redesign needed. Older models hit a wall much sooner. Why?
Start with training speed. Older recurrent networks read text one word at a time. Word ten had to wait for word nine. That's a traffic jam.
A transformer drops that rule during training. It processes every token in the sequence at the same time. One giant batch of math, instead of a long queue. And that math is almost all matrix multiplication.
Which is exactly what a GPU is built for. So thousands of chips can chew on one model together. Now the deeper reason. Researchers found scaling laws.
As you add parameters, data, and compute, the model's loss falls along a smooth, predictable curve. A power law. That predictability is gold. You can train small cheap models, fit the curve, and forecast how good a huge model will be.
Before spending millions on it. There's a recipe too. The Chinchilla result: for a given compute budget, scale parameters and training tokens together, roughly in step. A big model starved of data wastes money.
And the architecture cooperates. A transformer is one block, repeated. Want more capacity? Stack more blocks, make them wider.
Nothing else changes. So transformers scale because training runs in parallel, GPUs love the math, and quality improves on a curve you can predict. That's why bigger kept winning. Quick check now.
One question is coming up. Let's see if it clicked.