LoRA, dissected

What exactly is a weight matrix? Let's try to understand. In a simple way. A neural network eats a list of numbers and produces a new list of numbers.

A list like that is called a vector. The work happens in layers. A basic one, a dense layer, is just a grid of stored numbers. That grid is the weight matrix.

Each entry is a weight, or parameter. How is one output number made? Take one row of the grid, multiply each weight by the matching input number, and add everything up. That sum is a dot product.

Now repeat that for every row. Each row gives one output number, and together they form the output vector. That whole move is one matrix multiply. Tiny example.

Say the input has three numbers and we want two out. The matrix needs two rows and three columns. Six weights, two dot products, done. That's the shape rule.

Mapping an input of length k to an output of length d takes a d by k matrix, storing d times k numbers. Now scale up. GPT-3, the giant model this paper studies, has matrices twelve thousand two hundred eighty eight numbers wide. One square grid that size holds roughly one hundred fifty million weights.

Stack ninety six layers of this, and the whole of GPT-3 comes to about one hundred seventy five billion parameters. So hold this picture. A layer is one matrix multiply, and the model's knowledge lives in these grids. Next lesson, what it takes to change them.

Quick check now. One question is coming up. Let's see if it clicked.

How does fine-tuning work? Let's try to understand. In a simple way. A big language model comes pre-trained.

Someone already trained it on a huge pile of text, so its weight matrices are full of useful numbers. Fine-tuning means training it a bit more, on your data, for your task. Here is how one training step goes. The model reads an example and makes a prediction.

We compare it to the right answer, and the gap becomes one number, called the loss. Now the clever part. Backpropagation works backwards through the network and computes, for every single weight, which way to nudge it to shrink the loss. That direction is the weight's gradient.

Then gradient descent takes a tiny step. Every weight moves a little against its gradient. Do that thousands of times and the loss keeps falling. Notice what just happened.

Nothing was exempt. Every entry of every weight matrix, in every layer, got its own gradient and its own nudge. Full fine-tuning trains all of them. So the change you learn is exactly as large as the model itself.

For GPT-3, the update holds about one hundred seventy five billion numbers. Per task. And here is the sting. Those changed weights are your new model.

Adapt it to ten different tasks and you now store ten full-size copies. Nothing is shared. To be fair, this works, and it is the standard recipe. But every trainable weight drags hidden costs behind it, and that bill is where we go next.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the hidden memory bill of fine-tuning? Let's try to understand. In a simple way. Last lesson, fine-tuning nudged every weight.

For GPT-3, about one hundred seventy five billion numbers. Sounds like the whole bill. It is not. First hidden cost: every weight you train needs its gradient stored, that slope number from last lesson.

A second full copy of the model, sitting in memory. And who takes the step? An optimizer, the algorithm turning gradients into weight updates. The favorite for big models is called Adam.

Adam adapts each step. So for every trained weight it keeps two running averages: of the gradient, and of the squared gradient. These are its optimizer states. Two more copies.

Now count bytes. A thirty two bit number takes four bytes. So one trained weight costs four for itself, four for its gradient, eight for Adam. Sixteen total.

Twelve of those sixteen bytes exist only because the weight is being trained. Three quarters of the bill is training baggage, not model. At GPT-3 scale it explodes. The paper reports about one point two terabytes of GPU memory to fine-tune it with Adam.

Racks of hardware, not one card. Now flip it. Freeze a weight, and its gradient and averages never exist. It costs just its own four bytes.

Memory follows what you train, not what exists. So that is the real bill. The weights are the cheap part. The baggage costs several times more.

Shrink the trainable set, and the baggage disappears with it. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the cost of escaping full fine-tuning? Let's try to understand. In a simple way. Last lesson showed the bill: every trained weight drags gradients and optimizer states behind it.

The escape? Train fewer numbers. Before LoRA, three families tried it, and each paid a tax. First family: adapters.

Freeze every big weight matrix, and insert small new layers between the existing ones. Only the little inserts train. The tax is latency, waiting time. The inserts run in sequence, one after another, but GPUs thrive on parallel work.

A batch of one request? The paper clocked GPT-2 up to thirty percent slower. Second family: prefix tuning. Leave the network alone.

Learn a few artificial inputs instead: trainable vectors glued in front of your text, steering the frozen model. The tax is context. A model can only read so much at once, and every prefix slot steals a seat from your real task. More slots did not mean better: past a few hundred, scores dropped.

Third family: BitFit. Each layer also adds a short list of numbers to its output, its bias. BitFit freezes every matrix and trains only the biases. The tax is a ceiling.

Biases are a tiny, fixed set: no dial for more capacity. On GPT-3, BitFit beat full fine-tuning on one task, yet trailed by over two points on another. So: adapters cost time, prefixes cost space, BitFit caps capacity. The wish list is sharp now.

Few trained numbers, no added latency, no stolen context, a capacity dial. LoRA checks every box. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is the rank of a matrix? Let's try to understand. In a simple way. A weight matrix is a grid of numbers, rows times columns.

But here is the thing. Shape can lie about how much information is really inside. Picture a spreadsheet where one column is exactly double another. It fills space.

It looks like data. But it tells you nothing new. Here is the test. A row is genuinely new if you cannot build it by scaling and adding the others.

Mathematicians say linearly independent. Rank is the count of genuinely new rows. Try one. Three rows.

Row three is just row one plus row two. On paper, three rows. In truth, only two carry new information. So the rank is two.

Rank has a ceiling: the smaller of the two dimensions, rows or columns. Hit that ceiling, and the matrix has full rank. And low rank is the opposite. Every row is a mix of just a few underlying directions.

The grid looks big, but it is thin. Mostly echoes. Why care? Because thin means compressible.

If a few directions explain everything, you should not need all those numbers. Hold that thought for next lesson. Now the twist. The LoRA paper notes that the weight matrices of a trained network typically have full rank.

The weights themselves are not thin at all. But fine-tuning does not replace weights. It nudges them. So the sharper question is whether the change itself is thin.

That is where this course goes next. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is low-rank factorization? Let's try to understand. In a simple way. Last time we saw what makes a matrix low rank: its rows come from just a few shared directions.

Here is the payoff. A matrix of rank r splits exactly into two smaller ones. A tall thin one with r columns, times a short wide one with r rows. Multiply them, and every entry comes back.

Now count the storage. The full matrix stores row count times column count. The factored pair stores only r times, row count plus column count. When r is small, that changes everything.

Let's make it concrete. Take one square layer, four thousand ninety six wide. Stored in full, that is about seventeen million numbers. Now write it as a rank eight pair.

Each factor is four thousand ninety six by eight. Together, about sixty five thousand numbers. That is all. Same shape, same kind of matrix multiply, two hundred fifty six times fewer numbers.

That ratio is the whole trick. But here is the catch. The rewrite is only exact when the rank really is that small. Squeeze a full-rank matrix this way and you throw information away.

And remember the hidden bill. Every trainable number drags a gradient and optimizer states behind it. Train sixty five thousand numbers instead of seventeen million, and that whole bill shrinks with it. So one question decides everything.

Is the change that fine-tuning makes actually low rank? That is where the paper places its bet. Next lesson. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is intrinsic dimensionality? Let's try to understand. In a simple way. Last lesson gave us the trick: a low rank matrix stored as a thin product.

But it only pays off if fine tuning's update really is low rank. Why believe that? Here is a strange question. A model has millions of knobs, its weights.

To learn one new task, how many do you truly need to move? In twenty eighteen, Li and colleagues measured this. Their answer: far fewer than the raw count. Big models sit on a low intrinsic dimension, the count of directions learning genuinely requires.

Then in twenty twenty, Aghajanyan and colleagues found the same in pre trained language models. Squeeze training through a random projection into a much smaller slice of weight space, and the model still learns efficiently. Hu and colleagues read that as a clue. Their hypothesis: the change in weights during adaptation also has a low intrinsic rank.

Not the weights. Only the update fine tuning adds. That distinction matters. Trained weight matrices themselves typically have full rank, the paper notes.

The bet is only about the nudge a new task adds on top. And notice the word hypothesis. Nobody proved this. The paper states it as a bet, then runs experiments to test it.

That evidence comes later in the course. If the bet holds, last lesson's factorization applies to the update directly. Write the change as a thin pair, train only that, and lesson three's memory bill collapses. That construction is next.

Quick check now. One question is coming up. Let's see if it clicked.

How does the LoRA update work? Let's try to understand. In a simple way. Take one pretrained weight matrix, call it W.

The last lesson left us with a hypothesis: the change fine-tuning would make to W has low rank. So here is LoRA's move. Do not touch W at all. Freeze it.

Build the change as a separate little detour running right beside it. The detour is two thin matrices. A is short and wide, r rows. B is tall and skinny, r columns.

Here r is a small rank we choose. The update is B times A. Now walk one forward pass. The input vector x arrives, and the layer sends it into both paths at once.

Same x, twice. The frozen path computes W times x, exactly as it always did. Nothing from pretraining is lost. The detour hits A first.

A squeezes x down to just r numbers. Then B expands those r numbers back up to full output size. Both paths now hold a vector of the same length. Add them, entry by entry.

That sum is the layer's output, h. At training time, gradients flow only into A and B. W gets no gradient, so no gradient storage, and no optimizer states either. The hidden bill vanishes for every frozen weight.

And the detour is tiny. For a layer four thousand ninety six wide, rank eight means training sixty five thousand numbers instead of seventeen million. Quick check now. One question is coming up.

Let's see if it clicked.

How does LoRA's zero start work? Let's try to understand. In a simple way. So you bolt the detour, B times A, onto a trained model.

Both matrices need starting values. Pick badly, and step one scrambles what the model knows. The paper's answer is neat. A starts as small random numbers.

B starts as all zeros. Multiply them, and B times A is exactly zero. So on day one, the detour adds nothing. The first forward pass is identical to the pretrained model.

Training starts from a model that still behaves perfectly. Now, why not zero both? Here is the trap. If A and B are both zero, the gradient for each is zero too.

Nothing would ever move. With A random and B zero, the product is still zero, but B's gradient is not. B takes the first step, and the pair wakes up. There is one more knob.

The detour's output gets multiplied by alpha over r. Alpha is a fixed constant you pick, and r is the rank. Why divide by r? So the update keeps roughly the same strength when you change the rank.

Try rank four, then rank eight, without re-tuning everything. And Hu and colleagues barely tune alpha. With Adam, tuning it acts roughly like tuning the learning rate, the size of each training step. So they set alpha to the first r they tried.

So, the recipe is safe by design. Start exactly at the pretrained behavior, wake the detour gently, and keep its strength steady across ranks. Quick check now. One question is coming up.

Let's see if it clicked.

How does picking the right matrices and rank work? Let's try to understand. In a simple way. LoRA leaves two choices open.

Which matrices get a bypass? How small can rank r go? The paper answers with ablations: experiments changing one choice, everything else held fixed. Quick map.

Attention is where words look at each other. Each attention block holds four weight matrices: query, key, value, output. Those are the candidates. First ablation, on GPT-3.

Budget: eighteen million trainable parameters across ninety six layers. Rank eight on one matrix type, rank four on two, or rank two on all four. The result: everything on query alone, or key alone, clearly worse. Query and value together won.

Spreading small rank over more matrices beat one matrix with big rank. Second ablation: rank itself. They swept one, two, four, eight, sixty four. Here is the shock.

On a task turning questions into database queries, query plus value at rank one scored seventy three point four. Rank sixty four: seventy three point five. Sixty four times the capacity, almost nothing gained. Why?

They compared what rank eight and rank sixty four actually learned. The strongest direction matched in both. The extra directions barely overlapped, likely just noise accumulated during training. The authors add a caveat.

They do not expect tiny ranks to work everywhere. A task far from pre-training, say a new language, could need far more. So the recipe: give query and value the bypass, keep rank small. The update really is thin.

Quick check now. One question is coming up. Let's see if it clicked.

How does LoRA merging work? Let's try to understand. In a simple way. Remember the LoRA forward pass.

Input flows through the frozen weight, and through B times A, then the outputs add. Sounds like extra serving cost. But B times A is just a matrix, the same shape as the frozen weight. And adding matrices first, then multiplying, gives the exact same answer.

So before deployment, compute W zero plus B A once. One matrix, one multiply. Inference costs exactly what the original costs. Zero extra latency, by construction.

Adapters cannot do this. They stack extra layers with a bend inside, a nonlinear step, which blocks collapsing into one matrix. LoRA's bypass is pure addition, so it folds away. Switching tasks is arithmetic.

Subtract B A to recover the clean base, then add the next task's pair. Fast, with very little memory moving. So one frozen base can serve many specialists. Legal, medical, code.

Each specialty is just one small pair of matrices, a tiny file. How tiny? For GPT-3, a full fine-tuned copy weighs about three hundred fifty gigabytes. The rank four pairs on query and value?

About thirty five megabytes. Roughly ten thousand times smaller. But careful. The merged model is exactly the original size.

You still need the whole base to serve it. The saving is per-task storage, not serving memory. One catch the authors admit. Once merged, the model serves one task at a time.

To mix tasks in one go, skip the merge and pick each pair per request. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is LoRA's legacy? Let's try to understand. In a simple way. So, did the whole trick work?

Across four families, RoBERTa, DeBERTa, GPT-2, and GPT-3, LoRA scored on par with, or better than, full fine-tuning. On GPT-3, ten thousand times fewer trainable parameters, roughly three times less training memory, and each task's checkpoint at thirty five megabytes instead of three hundred fifty gigabytes. But mind the fine print. On some individual tasks, full fine-tuning still nudged ahead.

The honest claim is on par or better, not always wins. Caveat one, from the authors. Once the bypass is merged in, that copy speaks one task. Serving mixed tasks together means skipping the merge.

Caveat two. They only adapted attention matrices, leaving the rest to future work. And they warn tiny ranks won't fit every job, say a whole new language. Caveat three.

Picking which matrices to adapt is still a rule of thumb. And why adaptation works at all, the mechanism, stays open. Then the descendants. Twenty twenty three, QLoRA: shrink the frozen base to four bits per weight, train LoRA on top, and a sixty five billion parameter model fits on one forty eight gigabyte GPU.

Twenty twenty four, DoRA: split each pretrained weight into a size part and a direction part, and give only the direction a low-rank update. Steadier training, still no extra inference cost. So that's the story. A hypothesis about a few directions, tested on four model families, extended by a line of successors.

Freeze, add a thin bypass, merge. Quick check now. One question is coming up. Let's see if it clicked.