Architecture ideas that won

What exactly is multi-head attention? Let's try to understand. In a simple way. Read one sentence, and it holds many links at the same time.

Who acts on whom. Which noun a pronoun points to. What matches in tense. A model has to catch all of them together.

Attention lets each word look at the other words. But a single attention pattern is like one lens. It can lock onto just one kind of link. Aim it at grammar, and it misses the rest.

So the fix is simple. Run several attention patterns side by side. Each copy is called a head. A model might use eight of them, or many more.

Every head gets its own set of learned weights. So each one builds its own view of the very same sentence. One angle apiece, all from the same words. And the heads specialize.

One may track subject and verb. Another follows what a pronoun refers to. Another watches the word right next door. The best part.

All the heads run at the same moment, not one after another. So catching many relationships costs almost no extra time. Then the outputs are joined back together. The model stacks every head's result and blends them into one vector.

Many views, folded into a single answer. So, multi-head attention runs many lenses at once. Each head catches a different link, and together they see the whole picture. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is query, key, and value? Let's try to understand. In a simple way. In a sentence, every word needs context from the others.

Take the word it. To make sense, it must find the noun it stands for. Each word has to search the sentence. So attention gives every word three vectors.

A query, a key, and a value. Each comes from its own learned weight matrix. The query is what a word is looking for. Think of it as the word's question.

The word it asks, which noun do I point to? The key is what each word advertises about itself. A short label saying, here is what I offer. Every word puts one out.

Now compare the query to every key. Each pairing gives a score, how well the two line up. Softmax turns those scores into weights that add up to one. The value is the actual content a word carries.

Each value is pulled in by its weight, then all are summed. That blend becomes the word's new, context aware vector. Why three vectors, not one? Because the jobs differ.

What a word seeks, what it offers, and what it hands over are not the same. Splitting them lets the model learn each role. Picture a library. The query is your search request.

The keys are the labels on the shelves. The value is the book you carry home. So, query asks, key advertises, and value delivers. Match query to keys for the weights, then blend the values into new context.

Quick check now. One question is coming up. Let's see if it clicked.

How does sharing keys and values work? Let's try to understand. In a simple way. First, a quick recall.

In attention, every head builds a query, a key, and a value for each word. Queries ask, keys and values answer. When a model writes, it adds one word at a time. Instead of redoing the past, it saves every key and value to reuse.

That store is the key value cache. But that cache grows fast. Every layer, every head, and every past word keeps its own key and value. Moving all of it, step after step, slows the model down.

Here is the trick. Let several query heads share a single key and value, instead of each keeping its own. The queries stay separate. Only keys and values are pooled.

So the query heads are split into a few groups. Each group shares one key value pair. This design is called grouped query attention. Now the cache holds far fewer keys and values.

With less memory to move each step, the model generates each word much faster. And here is the clever part. Each query still asks its own question. Only what they read from is shared.

So quality stays close to full multi head attention. This is why large models like Llama use grouped query attention. It brings big speedups, while barely changing how well the model reads. So, sharing keys and values shrinks the cache.

Many query heads share fewer key value pairs, so generation runs faster at almost the same quality. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the speed versus quality attention tradeoff? Let's try to understand. In a simple way. Attention is powerful, but it is not free.

To answer, each attention head keeps a key and a value for every word so far. As the model writes, those keys and values pile up in a store called the key value cache. Longer text means a bigger cache to read for every new word. A model has many heads.

In the classic design, every head owns its own private keys and values. Great for quality, but the cache balloons. So here is the tension. More keys and values give sharper attention.

Fewer make the model lighter and faster. That is the speed versus quality tradeoff. Picture a dial. At one end, every head has its own keys and values.

At the other, all heads share one set. Quality here, speed there. Here is the trick that makes sharing safe. The queries stay separate, one per head, so each still asks its own question.

Only keys and values are shared. Most modern models sit in the middle. Heads are split into small groups, and each group shares one set of keys and values. Nearly full quality, far less cache.

Why does this pay off? During generation the model is starved for memory bandwidth, not math. Shrinking the cache is what makes it faster. So, keys and values are costly to store and move.

Share them wisely, and you trade a sliver of quality for a big jump in speed. Quick check now. One question is coming up. Let's see if it clicked.

How does multi query attention work? Let's try to understand. In a simple way. Normally, attention runs many heads at once.

Each head builds its own query, its own key, and its own value. Many heads means many separate keys and values. As the model writes, it saves every key and value. With lots of heads, that store grows large.

And moving all of it, word after word, is what slows things down. So here is the bold move. Keep all the query heads as they are. But give every one of them a single shared key, and a single shared value.

This is multi query attention. Picture eight heads. Full attention keeps eight keys and eight values, one set per head. Multi query keeps all eight queries, but only one key and one value for them to share.

Now the stored keys and values shrink by the number of heads. Eight times fewer here. With so little to move each step, the model writes each word much faster. There is a small cost.

Every query now reads that same single key and value. So the heads lose a bit of their own private view, and quality can dip a little. Multi query is the extreme end. One pair for everyone.

Grouped query sits in between, giving a few groups their own pair. PaLM and Falcon took the one pair route. So, multi query attention hands every query head one shared key and value. The cache shrinks, generation speeds up, and quality drops only a touch.

Quick check now. One question is coming up. Let's see if it clicked.

How does word order in a transformer work? Let's try to understand. In a simple way. A transformer reads every word in a sentence at the same moment.

All in parallel. On its own, nothing marks which word came first, or which came last. So to the attention layer, the input is just a bag of words, with no sense of sequence. The cat sat, and sat the cat, would look exactly the same.

But word order carries meaning. Dog bites man is a small story. Man bites dog is front page news. Same three words, a very different sentence.

The fix is simple. Give every position its own small vector. One vector for slot one, another for slot two, and so on down the line. Then add that position vector straight onto the word's own vector.

Not glued on beside it. Added in, so each word now quietly carries its place. Now the same word in two different spots gets two different vectors. Cat at the front is not cat at the end.

Attention can finally feel the order. Where do these position vectors come from? The first transformers used fixed sine and cosine waves. Many models since simply let the position vectors be learned.

So, attention on its own is order-blind. We add a position vector to every word, and suddenly the model can tell first from last. Quick check now. One question is coming up.

Let's see if it clicked.

How does encoding position with rotations work? Let's try to understand. In a simple way. Attention looks at every word at once.

On its own, it cannot tell which came first. The cat chased the dog reads like the dog chased the cat. So each word needs to know its position. One older trick was to add a position vector to each word.

Position one gets one vector, position two another. It works, but it bakes in a fixed address. Rotary encoding does something different. Instead of adding, it rotates each word's vector.

Picture the vector as a clock hand, and the word's position decides how far it turns. Word one turns by a small angle. Word two, twice as much. Word three, three times.

The further along a word sits, the more its hand turns. A real vector holds hundreds of numbers, not two. So they are grouped into pairs. Each pair acts like a tiny arrow, turned by that same position angle.

Here is the clever part. Not all pairs spin at the same rate. Some turn fast, others slowly. Fast hands catch nearby order, slow hands track distance across the sentence.

This spin is applied to the query and the key, just before attention compares them. The word's meaning stays the same. Only the angle carries where it sits. Rotary encoding writes position as a rotation, not an addition.

Spin each pair of numbers by an angle set by its position, and every word knows its place. Llama and many modern models use it. Quick check now. One question is coming up.

Let's see if it clicked.

How does rotating vectors preserve relative distance? Let's try to understand. In a simple way. Rotary encoding spins each word's query and key by an angle set by its position.

But what does attention actually gain from it? Attention scores two words with a dot product of query and key. That grows when the vectors point the same way. It reads the angle between them.

The crucial fact. Turning a vector never changes its length. And turning two vectors together leaves the angle between them untouched. Rotation is rigid.

Picture a query at position two, a key at position five. The angle between them shifts by exactly the gap, three steps. The score depends only on that distance. Here is the beautiful part.

Two words three apart score the same at two and five, or far down at two hundred. Absolute place cancels; only the gap remains. And relative distance is just what language cares about. One word sitting before another matters, not whether it lands in slot five or slot five hundred.

This buys something more. Because only the gap counts, a model trained on short passages can handle positions it never saw, as long as the distances stay familiar. Compare the older trick of adding a fixed vector to each slot. That baked an absolute address into the score.

Rotation drops the address, keeping pure distance. So, rotation keeps each vector's length and the angle between pairs. Spin query and key by their positions, and the score reads only the gap. Position becomes pure distance.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a transformer's thinking layer? Let's try to understand. In a simple way. Inside every transformer block, two steps happen in turn.

First attention, where words share information. Then a second layer, where each word thinks alone. Right after attention, each word holds a bundle of gathered context. But mixing is not thinking.

Something still has to process what was collected. So this layer visits every word one at a time. The same small network, applied to each position separately. A word looks only at itself here.

Its real name is the feed forward network. Just two simple layers with a nonlinearity wedged between them. Plain, but it does the heavy lifting. Step one is to expand.

The word's vector gets projected into a much wider space, usually four times as wide. Room to spread its features out. In that wide space a nonlinearity fires, a function like relu or gelu. It flips features on and off, and this is where the real reshaping happens.

Step three squeezes it back down to the original size. So the pattern stays simple. Expand, transform with the switch, then contract. Here is a surprise.

This layer quietly holds most of the model's weights, roughly two thirds. It acts like a memory, storing patterns learned from all that reading. So the thinking layer is a feed forward network run on each word alone. Widen, apply a nonlinear switch, narrow.

That is a transformer's private thinking. Quick check now. One question is coming up. Let's see if it clicked.

How does a mixture of experts work? Let's try to understand. In a simple way. Inside every transformer block sits one thinking layer, the feed forward network.

The same network runs on every word. But one network must handle everything. That is a lot to ask. One network must juggle grammar, code, history, and math, all in a single set of weights.

So here is the idea. Replace that one network with many copies, maybe a hundred. Each copy is an expert. Together, a mixture of experts.

But a word does not visit all of them. A small chooser, the router, picks which experts handle each word. How it picks waits for next lesson. Here is the trick.

Only a couple of experts wake up for each word. The rest stay asleep. Maybe two of a hundred run. That is sparse activation.

That split is the payoff. Total weights balloon into huge capacity. Yet each word touches only a couple of experts, so work per word stays small. Big brain, small bill.

Because each expert only sees some words, they specialize. One drifts toward code, another toward numbers, another toward everyday prose. Nobody assigns these roles. They emerge.

Routing happens fresh for every word. The word def can go to a code expert, while the goes to a common one. Each token is sent on its own. So a mixture of experts swaps one busy network for a room of specialists.

A router sends each word to a few, most sit idle. Huge capacity, cheap per word. Quick check now. One question is coming up.

Let's see if it clicked.

How does an expert router work? Let's try to understand. In a simple way. A mixture of experts holds many networks.

But each word should visit only a couple. So something must decide which ones. That job is the router's. The router is tiny, just a small layer in front of the experts.

The word's vector goes in, and the router decides where it should go. First it gives every expert a score. It reads the word's vector and asks how well each expert fits. One number per expert.

Higher means a better match. Then those raw scores pass through a softmax, turning them into weights that add up to one. Now each expert has a share, how much it matters here. But we do not run all of them.

The router keeps only the top few, often just two. The rest are ignored. That is what keeps it cheap. The two chosen experts each process the word.

Their outputs are blended, weighted by those gate values. A bigger share pulls the result more. One answer comes out. Nobody wrote these rules by hand.

The router is learned, trained alongside the experts. Over time it discovers which expert suits which word, on its own. There is a catch. Left alone, the router may send almost every word to a few favorite experts, leaving others idle.

So a balancing pressure spreads the load evenly. So the router is a tiny learned layer. It scores every expert, turns those into weights, and keeps the top few. Cheap, and balanced.

Quick check now. One question is coming up. Let's see if it clicked.

How does a model that reads one word at a time work? Let's try to understand. In a simple way. Attention reads the whole sentence at once.

Every word compares itself to all the others. Powerful, but the work grows fast. Twice the words, four times the cost. There is an older idea.

Read the text one word at a time, left to right, the way a person reads down a page. As you read, you keep a running summary in your head. The model keeps one small vector, its state, the memory of everything so far. Each new word updates that state.

The new state is the old one blended with the word just read. The same step, repeated for every word. This is cheap. One step per word, so the total work grows in a straight line.

And the state stays one fixed size, however long the text runs. When the model writes new words, it never rescans a growing history. It just carries that little state forward. So each word comes out fast, on steady memory.

But there is a catch. Everything must squeeze through one fixed vector. A narrow gate, so faint details from far back can quietly fade. This design has a name.

A recurrent model, or a state space model. An old loop, now rebuilt with fast modern math. Mamba is the famous example. So, read one word, update the state, move on.

Linear cost, constant memory, and a single running summary doing all of the remembering. Quick check now. One question is coming up. Let's see if it clicked.

How does Mamba work? Let's try to understand. In a simple way. Mamba reads one word at a time, keeping a small memory vector, its state.

That state is a running summary of everything read so far. Older versions had a flaw. The update rule was fixed, the same blend for every word. So a filler word and a key name were treated alike.

Mamba's fix is one word. Selective. Now the update depends on the very word being read. That decision is two dials, computed from the word itself.

One says how much old memory to forget. The other says how much of this new word to write in. So on a filler word, it barely touches the state. On an important word, a name or a fact, it writes that in strongly.

It can even wipe the memory when the topic turns. This is content based gating. Nobody hand codes which words matter. The dials are learned, so the model teaches itself what to hold and what to drop.

There is a catch. Making the update depend on the word breaks the old fast shortcut. Mamba wins it back with a hardware aware parallel scan, so training stays quick. The payoff is the best of both.

Cost still grows in a straight line, like reading once. Yet the memory is smart, so it stays strong on very long text. So, Mamba carries one small memory, and updates it selectively. Forget the filler, remember what matters, learned from data.

Linear cost, with a memory that chooses. Quick check now. One question is coming up. Let's see if it clicked.

How does positional interpolation work? Let's try to understand. In a simple way. A model is trained with a fixed context window.

Say two thousand positions. Every slot gets a position code, its address in the line. Now feed it a much longer document. Suddenly it meets positions far past that trained edge.

Addresses it has never seen before. The naive move is to just keep counting outward. But those faraway codes are unfamiliar, and attention falls apart. The model breaks.

Positional interpolation flips it. Instead of pushing positions outward, it squeezes them back in. The longest new position lands right at the old edge. Concretely, to double the window, halve every position number.

Position four thousand becomes two thousand. Back inside the trained range. Now the positions sit closer together. Fractional values slot between the whole numbers it knew.

Denser, but all within the safe band. Here is why interpolating wins. Inside its trained range the model behaves smoothly. Outside it, wildly.

Interpolation keeps everything inside. One catch. The spacing is new, so the model needs a short fine tune. About a thousand steps teaches it the tighter grid.

Cheap. The payoff. The very same weights now read text twice or four times as long. A tiny tune buys a much bigger window.

So, positional interpolation shrinks the positions instead of stretching them. Keep every address inside the trained range, add a short tune, and the context grows. Quick check now. One question is coming up.

Let's see if it clicked.

How does sliding window attention work? Let's try to understand. In a simple way. In normal attention, every word looks at every other word.

To understand one word, it scans the whole sentence. Nothing gets left out. That thoroughness has a price. Ten words make a hundred links.

A hundred words make ten thousand. Double the length, and the work jumps four times. The fix is simple. Give each word a small window.

Instead of the whole sentence, it looks only at a handful of nearby neighbors. Say the window is five hundred. Each word attends only to the five hundred words just before it. A fixed band that slides along.

Now the cost grows in a straight line. Each word does the same fixed work, no matter how long the text gets. Quadratic becomes linear. But wait.

If a word only sees its neighbors, how does it ever reach something far away? Here is the trick. Stack many layers. In each layer a word blends in its neighbors.

But those neighbors already blended in theirs. So reach spreads, window by window, the deeper you go. Some designs also add a few global tokens. These special slots look at everything, and everything looks at them.

A shortcut for information that must travel far. So, windowed attention swaps the whole sentence for a small neighborhood. Linear cost per word, and depth quietly restores the long range reach. Quick check now.

One question is coming up. Let's see if it clicked.