LoRA, QLoRA and friends

Why is full fine-tuning a large AI model so incredibly expensive? Let us look at what actually happens inside your hardware. Imagine a standard seven billion parameter model. Just loading those base weights into memory takes about fourteen gigabytes.

That is before we even start training. Training requires a forward pass. The model looks at your data and calculates intermediate values called activations. We have to store every single one of them.

Then comes the backward pass. The model calculates gradients to figure out how it should improve. That creates a brand new number for every single parameter. But the real memory hog is the optimizer.

This is the algorithm that decides exactly how to update the model weights safely. Most full fine-tuning uses an optimizer called Adam. Adam is highly effective, but it is very greedy. It keeps two extra tracking numbers for every parameter.

Let us do the math. For every original weight, you now have a gradient and two optimizer states. One parameter becomes four separate numbers in memory. Your compact fourteen gigabyte model just exploded.

Full fine-tuning now requires well over sixty gigabytes of memory just to run the process. A single top tier consumer graphics card usually has twenty four gigabytes of memory. You would need multiple expensive cards just to hold everything at once. This massive memory footprint is exactly why full fine-tuning costs so much money.

You are paying for the sheer volume of numbers being constantly tracked. Quick check to see if you caught the biggest memory hog.

What exactly is a frozen weight? Let's try to understand. In a simple way. During training, every weight we want to update carries baggage.

It needs a gradient, and the optimizer keeps two more tracking numbers for it. So one weight quietly becomes four numbers in memory. On a model with billions of weights, that hidden baggage is what fills up your graphics card. Here is the trick.

We freeze most of the weights. Freezing just means we mark them as not trainable. We promise never to update them. When a weight is frozen, the backward pass skips it.

No gradient gets calculated for it, and no gradient has to be stored. The optimizer ignores it too. It keeps no tracking numbers for a weight it will never change. That baggage simply disappears.

The weight itself stays in memory. The forward pass still reads it to make predictions. It is present, but read only, like a locked page. So where does the learning happen?

We add a small set of fresh weights, and we leave only those trainable. Now only that tiny group carries gradients and optimizer numbers. The heavy baggage shrinks from billions of weights down to a sliver. The result is striking.

The same huge model now trains inside a fraction of the memory. Often it fits on a single card. So freezing weights means locking the base model in place, and paying the training cost only for the few weights that still learn. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is low-rank adaptation? Let's try to understand. In a simple way. Fine-tuning wants to change a big weight matrix in the model.

That change is itself a matrix, the same giant shape. Storing the whole change is what we avoid. So here is the key word. Rank.

A matrix's rank is the number of truly independent patterns inside it. How many rows carry new information, and how many just echo others. In a full-rank matrix, every row points its own way. Nothing repeats.

To describe it, you need every single number. In a low-rank matrix, the rows are blends of just a few base patterns. Most of the grid is redundant, so those few rebuild all of it. Here is the surprise.

The update fine-tuning needs is low-rank. The change is small and focused, so it leans on just a few directions. So you never store the giant grid. You keep only those few patterns, a thin slice of numbers, that stand in for the full update.

The number of patterns you allow is the rank, which you pick. A tiny rank is cheap but can miss the skill. A bigger one captures more, at a small cost. And why is the update low-rank?

Adapting to one narrow task does not rewire the model. It nudges it along a few directions, and that nudge is exactly what low-rank holds. So low-rank adaptation means this. The change you need is simple, so you describe it with a few patterns instead of a whole giant grid.

Quick check now. One question is coming up. Let's see if it clicked.

How does splitting one big matrix into two work? Let's try to understand. In a simple way. Fine-tuning wants to add a change to a weight matrix.

That change is one big grid, the same shape as the weight. Call its size a thousand rows by a thousand columns. Count the numbers inside it. Rows times columns.

A thousand by a thousand is a million numbers, for one layer. Storing that is exactly what we want to dodge. So here is the trick. Never build that big grid.

Instead, write it as two smaller matrices, multiplied together. Call them B and A. The shapes are the whole point. B is tall and thin.

A is short and wide. They meet at a tiny shared number in the middle. That number is the rank. Now multiply B by A.

Out comes a grid of the full original size, a thousand by a thousand. The product stands in for the big change. But count what you actually store. B holds a thousand times the rank.

A holds the rank times a thousand. Add them, nothing more. Let the rank be eight. Each piece holds eight thousand numbers, sixteen thousand in total.

Against a million, that is a tiny sliver. Those two skinny matrices are the whole adapter. Only they get saved and trained. The giant change is never formed or stored at all.

So splitting one matrix into two means this. One huge update becomes a thin pair whose product rebuilds it, for a fraction of the numbers. Quick check now. One question is coming up.

Let's see if it clicked.

How does LoRA training work? Let's try to understand. In a simple way. Start with a finished model, its huge weights all frozen.

Beside one of them sits the adapter, a tiny pair of matrices. That pair is the only thing we train. At the start, the adapter adds nothing. One of its two matrices begins at all zeros.

So on step one, the model acts exactly like the untouched original. Now we show it examples of the new skill. A prompt goes in, the model answers, and we compare that answer to the one we wanted. The gap between them is the error.

Shrinking that gap, example after example, is the entire goal of training. To shrink it, the error flows backward through the network. The frozen weights are locked, so nothing sticks there. The signal lands only on the adapter's two matrices.

There, gradient descent nudges each adapter number a small step in the direction that lowers the error. The giant base never moves. Answer, compare, nudge, repeat. Thousands of times over the task's examples.

With every pass the adapter drifts further from zero, and the skill takes shape inside it. When training ends, that pair holds a small correction. Laid on top of the frozen weights, it steers the model toward the new task. The skill now lives in those few numbers.

So the base stays general and untouched, while a tiny trainable adapter learns the one new thing you asked for. Quick check now. One question is coming up. Let's see if it clicked.

How does adding adapter weights during inference work? Let's try to understand. In a simple way. A finished LoRA model has two parts sitting side by side.

The frozen base weight, and the small adapter next to it. When you run the model, your input flows into both at once. The base does its usual job. The input passes through the big frozen weight and produces its normal output.

This is the general answer the original model always gave. The same input also takes a short side trip through the adapter. Down through the first small matrix, then up through the second. Out comes a tiny output, a correction.

Now the key step. The base output and the adapter output are added together, number by number. That sum is what the layer actually passes on. Here is the neat part.

Adding those two outputs gives the exact same result as one combined weight would. The base plus the adapter's change, only computed on separate paths. Because the base is never touched, the adapter stays a separate little piece. So you can pop one adapter off and snap another on, keeping a single base in memory while serving many skills.

There is a small price. Every adapted layer now runs two matrix steps instead of one. A touch more compute at run time, though the extra work is tiny. So at inference the base and the adapter run in parallel, and their outputs are added.

Two paths in, one answer out. Quick check now. One question is coming up. Let's see if it clicked.

How does merging adapters for deployment work? Let's try to understand. In a simple way. A served LoRA model usually keeps two pieces side by side.

The frozen base weight, and the small adapter next to it. Every input runs through both, then their outputs are added. That second path is not free. On every single request, each adapted layer does an extra matrix step.

One user, barely noticed. A million users, and that little tax repeats a million times. So here is the trick. Instead of paying that tax on every request, do the adapter math just once, before the model ever goes live.

Merging is that one time step. Multiply the two small matrices back into their full size change. Then add that change into the base weight, number by number. Now you have a single new weight.

The adapter is gone. What is left is one weight, the exact same shape as the original. Each layer is back to a single matrix step, with nothing sitting beside it. This is the payoff.

At serve time the merged model runs at the plain base model's speed. No side path, no extra work, on every token it produces. But the skill is now baked in. You cannot pop the adapter back off, and each merged model is a full size copy to store.

You trade flexibility for raw speed. So merging folds the adapter in once, and you serve at full speed. Keep it separate to swap many skills. Merge when you serve one skill, fast.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the memory cost of a huge base model? Let's try to understand. In a simple way. LoRA already shrank the part you actually train down to a tiny adapter.

So the training math is cheap. But one big thing is still sitting in memory the whole time. The base model itself. Freezing the base stops its gradients, but it does not remove the weights.

To run even one forward pass, every base weight has to be loaded and held in memory. So how heavy is that? Each weight is a number. In the common sixteen bit format, one number takes two bytes.

Multiply the parameter count by those two bytes, and you get the size. Take a seven billion parameter model. Seven billion numbers, two bytes each, is fourteen gigabytes. That is the base weights alone, before the adapter adds anything.

And this grows straight with size. Double the parameters, double the memory. A seventy billion parameter model needs one hundred forty gigabytes, just to sit there. But your graphics card has a fixed amount of memory.

Even a large single card holds only about twenty four, or eighty gigabytes. That is a hard wall. So the base weights alone can overflow the card, before the adapter adds a single number. LoRA shrank the training cost, but it never touched this weight floor.

So the frozen base still costs parameters times bytes, and that fixed weight memory is the real ceiling. The only way down? Make each number smaller. That is where quantization comes in.

Quick check now. One question is coming up. Let's see if it clicked.

How does four-bit quantization work? Let's try to understand. In a simple way. A model is a giant pile of numbers called weights.

Each weight sits in memory, and normally every single weight takes sixteen bits of space. A big model has billions of these weights. Multiply it out, and they fill many gigabytes. That is why they need large, expensive cards.

Quantization asks a simple question. What if each weight used fewer bits? Say four bits instead of sixteen. Same weights, but a smaller box for each one.

Here is the catch. Four bits can only spell sixteen different values. So you get a short menu of just sixteen levels to choose from. So each real weight is rounded to the nearest level on that menu.

Like snapping a smooth dial to sixteen fixed notches. Close, but no longer exact. But weights vary in size. So they are split into small blocks, and each block keeps one scale factor that stretches the sixteen levels to fit its range.

Now the payoff. Sixteen bits per weight becomes four. That is four times less memory. A model that needed a huge card can suddenly fit on a small one.

Nothing is free. Rounding adds a tiny error to every weight. But large models are forgiving. A big model in four bits still beats a small model at full precision.

So four-bit quantization keeps every weight, but stores each on a coarse grid of sixteen levels. Far less memory, for only a little lost precision. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is NormalFloat four? Let's try to understand. In a simple way. Start with plain four-bit quantization.

It rounds each weight onto sixteen levels. And those levels are spaced evenly, like a ruler with equal steps. But here is the thing. The weights are not spread out evenly.

Most of them sit close to zero, and only a few are large. They form a bell curve. So even spacing wastes levels. Out in the empty tails, levels sit where almost no weights live.

And near zero, where they crowd, the steps are too coarse. NormalFloat fixes this. It places the sixteen levels to match the bell curve. Packed tightly near zero, and spread apart out in the sparse tails.

How? Each level is chosen so it covers an equal slice of the weights. Sixteen buckets, each holding the same share. This trick is called quantile quantization.

The result is dense levels in the crowded middle, and sparse levels near the empty edges. Now almost every weight lands right next to a level. That is the name. Normal, because the weights follow a normal bell curve.

Float four, because it still fits inside four bits. And zero maps exactly. The payoff is lovely. Same four bits, same memory as plain quantization.

But the rounding error shrinks, so the model stays more accurate. Better precision, for free. So NormalFloat four spends its sixteen levels wisely. It bends the grid to the shape of the weights.

Same size, but less error. Quick check now. One question is coming up. Let's see if it clicked.

How does double quantization work? Let's try to understand. In a simple way. Four-bit quantization already shrank the weights.

The weights were split into small blocks, and each block got its own scale factor. A number that stretches the levels to fit that block. But here is the catch. A scale factor is itself a number, stored in full precision.

Thirty two bits, sitting next to the tiny four-bit weights. The blocks are small, maybe sixty four weights each. So there are millions of scale factors. All that full precision adds up to about half a bit for every weight.

Now the clever idea. A scale factor is just a number. So it can be quantized too, the same trick we used on the weights, turned back onto the scales. So we run a second round.

Gather all the scale factors, and squeeze each from thirty two bits down to eight. That is where the name comes from. Quantization, twice. This second round needs its own scale factor, one for a whole group of scales.

But there are far fewer, so that cost is tiny. The overhead drops from about half a bit per weight to roughly a tenth. On a huge model, that quietly saves a few gigabytes. And it is nearly free.

Scale factors are helpers, not the weights themselves. Rounding them a little barely touches what the model can do. So, double quantization quantizes the scale factors of the quantized weights. A second squeeze on top of the first, for a bit more room.

Quick check now. One question is coming up. Let's see if it clicked.

How does a paged optimizer work? Let's try to understand. In a simple way. During training, the memory a run needs is not flat.

Most of the time it sits low. But every so often, it suddenly jumps. One awkward batch causes that jump. A very long sequence needs far more scratch memory for a moment, then settles down.

Here is the danger. The card has a hard memory ceiling. Cross it for even one instant, and the run dies. Out of memory.

So paged optimizers borrow a trick from your operating system. The same trick that lets a laptop juggle more than its memory should hold. When a computer's memory fills, it quietly moves the pages you are not using out to disk. When you need them again, it brings them back.

The optimizer keeps helper numbers for every weight it trains. Its state. That state lives in special memory, shared between the card and the computer. Now when a spike hits, the state is pushed over to the computer's memory.

The card frees up room, and the spike fits. No crash. Later, when the update step needs the state, it is paged back onto the card. A tiny pause for the transfer, not a dead run.

Why move the state, and not the weights? The state is only used at the update step. The rest of the time it sits idle. So a paged optimizer lets its memory spill safely onto the computer.

Spikes get absorbed, instead of killing the run. Quick check now. One question is coming up. Let's see if it clicked.

How does DoRA work? Let's try to understand. In a simple way. Start with a single weight in the model.

You can describe it two ways at once. How strong it is, its magnitude. And which way it points, its direction. So one weight hides two facts.

A length, and a heading. Pull them apart, and you can talk about each one on its own. Now recall plain LoRA. It adds one small low-rank update on top of the weight.

But that single change tugs the length and the direction together. They move as one. Full fine-tuning does not behave that way. It can stretch a weight a lot while barely turning it.

Or turn it without changing the length. The two move freely. So here is DoRA's idea. Before training, split every weight on purpose.

One part holds its magnitude, one part holds its direction. Then update the two parts with different tools. The direction gets a LoRA adapter, a low-rank change. The magnitude becomes its own small, trainable number.

At use time, put them back together. Take the adjusted direction, then scale it by the trained magnitude. That rebuilds the final weight. Because the two are trained apart, length and direction can now move on their own.

Much closer to full fine-tuning. Yet you still add only a tiny slice of numbers. So DoRA splits each weight into magnitude and direction, then tunes them separately. One knob for strength, a low-rank adapter for heading.

Quick check now. One question is coming up. Let's see if it clicked.

How does combining multiple LoRAs work? Let's try to understand. In a simple way. Recall what one LoRA adapter is.

A small pair of matrices that produces a tiny change to the base weights. Call that change a delta. Now, people train many adapters. One for a painting style, one for a certain character, one for a language.

Each is its own little delta, apart from the frozen base. Often you want two of them at once. The painting style, and the character, together, on the very same base model. Here is the neat part.

Each adapter is just a delta you add. So to combine them, add both deltas to the base. Base, plus delta one, plus delta two. You can also give each a weight.

A dial for how much of this style, how much of that skill. Like sliders on a mixing board, blending the adapters. Once you pick the blend, you can fold the summed deltas into the base weights. That leaves a single model.

No extra adapters to carry, and full base speed. But there is a cost. The adapters often edit the same weights. Their changes can collide, pulling in different directions.

This is called interference, and it can blur each skill. And a merged blend is baked in. To change the mix, you must merge again. Stack too many, and every skill gets washed out.

So, combining LoRAs means adding their deltas, each with its own weight, into one base. Powerful, but watch for interference when they overlap. Quick check now. One question is coming up.

Let's see if it clicked.

How does LoRA's alpha parameter work? Let's try to understand. In a simple way. Recall how a LoRA adapter works.

Two small matrices, B and A, multiply to build a change to the frozen base. Call that raw change a delta. But that delta is not added at full strength. Before it reaches the base, it gets multiplied by a scaling number.

Alpha is what sets that number. Think of alpha as a volume knob on the adapter. Turn it up, and the learned change speaks louder. Turn it down, and the base barely hears it.

Here is the exact rule. The scale is alpha divided by the rank. So the base becomes W plus, alpha over r, times B times A. Why divide by the rank?

A bigger rank stacks more numbers into the delta, making it naturally larger. Dividing by r cancels that, so the strength stays steady. That is the real trick. You can raise the rank for more capacity, without secretly making the update stronger.

The two knobs stay separate. In practice, people often set alpha to twice the rank, or simply equal to it. They pick a ratio, then leave it and tune other things. One more thing.

Alpha is not learned during training. You choose it up front, like the learning rate. It shapes how loud the adapter ends up. So, alpha over rank is a fixed dial on the adapter's volume.

It scales the delta before adding, and holds the strength steady as rank changes. Quick check now. One question is coming up. Let's see if it clicked.