Quantization, properly

What exactly is the memory cost of a single weight? Let's try to understand. In a simple way. A trained model is a huge pile of weights.

Weights are the values it learned during training. A small model has millions. A big one has billions. And each weight is nothing fancy.

It is simply a number. A decimal like zero point one three. That is all a weight is. But a number is not free.

The computer must set aside a little room in memory to hold it. That room is the weight's memory cost. By default, each weight gets thirty two bits. Eight bits make a byte, so that is four bytes per weight.

This format is called float thirty two. You do not always need that much. Use sixteen bits, and each weight costs two bytes. Same weight, half the memory.

Four bytes sounds tiny. But you never store one weight. You store all of them at once. Multiply four bytes by every weight.

A model with seven billion weights, at four bytes each, is twenty eight gigabytes. Just to hold the numbers. Big models do not fit on small cards. Here is the key idea.

Shrink the room each weight takes, and the whole model shrinks with it. Fewer bytes per weight means a smaller model. That is the door quantization opens. One weight is a number, and its cost is the bytes that store it.

Four bytes is normal. Multiply by billions, and the bytes decide if the model fits. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is quantization? Let's try to understand. In a simple way. A model is packed with numbers.

Each can sit almost anywhere on a smooth line of values. Between any two, there is always another. The possibilities feel endless. Quantization makes a trade.

Instead of allowing every value, we keep only a short list of allowed ones. A small menu of numbers we agree to use. Picture that menu as evenly spaced marks along the line. Only those marks count now.

Every gap between them is off the table. So what about a number that lands in a gap? It moves to the nearest mark. We call that rounding, or snapping to the grid.

Here is the heart of it. Many different starting numbers all snap to the same mark. A whole cluster folds into one. Many numbers become fewer numbers.

Each number moves only a little. So the result is close to the original, but not exactly equal. That small gap is the rounding error. You already do this.

Rounding a price to the nearest dollar. Sorting clothes into small, medium, and large. Many exact inputs, a few tidy buckets. How many marks should there be?

With very few, many values pile onto each one, and more detail is lost. With more marks, the copy stays truer. So quantization is a mapping. A huge range of numbers, sent onto a short menu, each rounded to its nearest neighbor.

Many, into fewer. Quick check now. One question is coming up. Let's see if it clicked.

How does turning a weight into a whole number work? Let's try to understand. In a simple way. A model's weights are messy decimals.

Numbers like zero point one three, or minus zero point seven two. Writing out every digit is expensive to store. But computers love whole numbers. A small whole number, like forty two, needs very little memory.

So swap each messy decimal for a plain whole number. The trick is one shared value called the scale. Think of it as the size of a single step. Every weight becomes a whole number of these steps.

To convert a weight, divide it by the step size, then round to the nearest whole number. That whole number is the only thing we store. To use the weight again, we multiply that whole number back by the step size. Out comes a decimal, close to the original, though not exactly it.

So the values we can land on form an even ladder. Rung, rung, rung, each one step apart. Every real weight snaps to its nearest rung. And that step size is shared.

One scale covers a whole block of weights. So we keep many tiny whole numbers, plus just one decimal for the step. Snapping to the nearest rung is never perfect. The little gap left behind is rounding error.

Smaller steps mean less error, but the whole numbers climb higher. So a weight becomes a whole number of steps, plus one shared step size. Tiny integers replace fat decimals. That is speaking in whole numbers.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the giant impact of tiny outliers? Let's try to understand. In a simple way. To shrink a weight to fewer bits, we pick a range that covers every value, then split it into a fixed number of evenly spaced levels.

The gap between two neighboring levels is the step size. Every weight snaps to the nearest level, so that step is exactly how much detail you can keep. Now here is the key. Almost all the weights sit in a tight little cluster close to zero.

Thousands of them, packed together. But every so often, one weight is enormous. Far bigger than the rest. We call it an outlier.

Just a handful out of thousands. The range must cover that outlier. So the instant it appears, the whole range stretches wide open just to reach it. But the number of levels never changed.

Spread those same levels across a much wider range, and every step suddenly turns far coarser. Now the tight cluster near zero is in trouble. Those weights are packed closer together than one single step. So they all snap to the same few levels.

So one rare outlier flattened thousands of normal weights into nearly the same number. A tiny few ruined the precision of the many. This is why a naive shrink can wreck a model's accuracy. Good quantization is really the art of taming these rare outliers.

So, a few extreme values force a wide range. A wide range means coarse steps. And coarse steps crush everything sitting near zero. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the extreme challenge of four-bit numbers? Let's try to understand. In a simple way. What four bits even means.

Each bit is one yes or no. Four together give two times two times two times two. That is sixteen. Only sixteen.

So a four-bit number can only be one of sixteen values. Picture sixteen pegs on a line. Every weight must land on a peg. Nothing in between is allowed.

Compare that. The original weights are thirty-two bit floats, billions of possible values. Eight-bit gives two hundred fifty-six levels. Four-bit gives sixteen.

A brutal drop. With only sixteen pegs, the gaps are wide. A weight between two pegs snaps to the nearest one. How far it moves is the rounding error, now a large jump.

One weight nudged is harmless. But a model has billions of them, across many layers. Each rounding error feeds the next layer, and they pile up. At eight bits the pegs sit close, so the model barely notices.

At four bits, plain rounding can wreck its answers. Same idea, far less room. So why try? Because four bits is half the memory of eight.

A model that needed two cards might now fit on one. The prize is huge. That is the tension. Sixteen pegs is too few for naive rounding, yet too valuable to skip.

So four-bit needs smarter tricks, not just nearest-peg snapping. So, the challenge is this. Four bits means sixteen values, wide gaps, piled-up error. Big memory savings, but only if you round with care.

Quick check now. One question is coming up. Let's see if it clicked.

How does shrinking a model without full retraining work? Let's try to understand. In a simple way. Picture a model that is already trained.

Every weight is a full sized number. To shrink it, we store each weight in fewer bits. But rounding always loses a little accuracy. One fix is to train the model again with the rounding baked in.

It slowly learns weights that survive being squashed. This is called quantization aware training. But that path is brutal. It needs the full training setup, the original data, and many costly passes over billions of weights.

So here is the smarter idea. Take the finished model and shrink it directly. No going back to training. This is called post training quantization.

The laziest version just rounds every weight to the nearest allowed value. Quick, but at low bit widths the errors pile up and the model gets worse. The smarter version feeds the model a small handful of example inputs. A few hundred, not the whole dataset.

It just watches what numbers flow through each layer. Then it picks the rounding and scales that keep each layer's output close to the original. Errors that matter get corrected. No labels, no backward passes.

So a model that would take days to retrain can be quantized in minutes on a single GPU, while keeping almost all of its accuracy. So, smarter shrinking skips retraining. Use a little calibration data to round wisely, and a finished model gets small without going back to school. Quick check now.

One question is coming up. Let's see if it clicked.

How does GPTQ think, layer by layer? Let's try to understand. In a simple way. Start with a trained model, a tall stack of layers holding billions of weights.

Rounding them all at once, while tracking how each change ripples to the output, is a hopeless tangle. So GPTQ refuses to solve the whole thing at once. It splits the network into layers and handles them one at a time, first layer to last. Zoom into one layer.

For now, forget the rest of the network. GPTQ cares only about this layer's numbers, nothing downstream. Here is the trick. Push a few example inputs through the original layer and note what comes out.

The quantized layer should give almost the same output. Match the layer, not the final answer. Because each layer carries its own small target, GPTQ never runs a backward pass through the whole model. No labels, no end to end training.

Layers are done in order, then locked. Inputs for the next layer come from the ones already quantized, so each sees the real, slightly drifted signal and absorbs it. That is the payoff. A billion weight tangle becomes a stack of tiny problems.

Each is easy, so the model is quantized in minutes on one card. Full retraining reasons about the entire model at once. GPTQ gives that up on purpose, trading one global goal for many fast local ones. So, GPTQ thinks layer by layer.

Split the model, match each layer's own output, and a giant problem becomes many small ones. Quick check now. One question is coming up. Let's see if it clicked.

How does minimizing error one layer at a time work? Let's try to understand. In a simple way. Quantizing a weight means rounding it to the nearest allowed value.

Do that to every weight alone, and each looks close. But tiny errors pile into a wrong answer. What actually leaves a layer is its output. A list of numbers it hands to the next one.

That output is all the rest of the model ever sees. So the real goal is not matching each weight. It is keeping the layer's output as close as possible to the original. That gap is the reconstruction error.

To measure that gap, you need real inputs. So a small batch of real text runs through, once with the full weights, once with the quantized ones. Line up the two outputs and look at the difference. Square it, add it up.

That single number is the error to shrink. Here is the trick. You do not fix the whole network at once. You take one layer, shrink its error, lock it, then move on.

Why local? The full network is far too tangled to solve in one shot. But one layer is a tidy little problem you can solve exactly. So instead of rounding blindly, the method hunts for the quantized weights that keep this output error as small as possible.

That is the whole point. So, quantize by output, not by weight. One layer at a time, feed in real data, shrink the reconstruction error. Small changes, faithful answers.

Quick check now. One question is coming up. Let's see if it clicked.

How does GPTQ save the important weights? Let's try to understand. In a simple way. Quantizing a weight means snapping it to the nearest value on a coarse grid.

That snap is never perfect. It leaves a tiny leftover, the rounding error. The lazy way rounds every weight on its own and moves on. Each looks close.

But across a whole layer, those little errors pile up, and the output drifts. GPTQ refuses to move on so fast. It rounds just one weight, then measures the exact error that snap created. It knows precisely how far off it landed.

Here is the clever part. It nudges the weights it has not touched yet, just enough to cancel that error. The mistake is not dropped. It is passed forward.

So GPTQ sweeps across the row one weight at a time. Each weight it locks hands its error to the free weights still waiting on the right. How big should each nudge be? GPTQ keeps a sensitivity map, built from a small sample of real data.

It says which weights the output leans on the most. Now the important weights are saved. Their errors get pushed onto the weights that barely matter, so the directions the layer truly cares about stay sharp. The payoff.

Weight by weight, these corrections keep the layer's output nearly identical to the original, even squeezed all the way down to four bits. So GPTQ does not just round. It rounds, measures the error, and pays it forward, protecting the weights that matter most. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is a salient weight? Let's try to understand. In a simple way. Quantization rounds every weight to the nearest low bit value.

So far, it has treated all of them the same. But here is the surprise. The weights are not equally important. Damage one weight and the model barely notices.

Damage another, and the answers fall apart. It turns out only a tiny slice carries most of the impact. Roughly one in a hundred. These are called the salient weights.

The other ninety nine percent are almost filler. So which ones are they? The obvious guess is the biggest numbers. But that guess is wrong.

A large weight is not automatically an important weight. What matters is the input each weight sees. A weight that multiplies a large incoming activation has a big say in the output. This is why the trick is called activation aware.

And those large activations are not spread out. They pile up in just a few input channels. The weights reading those channels are the ones that truly matter. So here is the payoff.

Keep that one percent in full precision, and round the rest hard. You throw away almost nothing, yet the model stays sharp. Protecting a single percent of the weights recovers most of the accuracy that plain rounding would have destroyed. The model almost forgets it was ever shrunk.

So, not all weights are equal. A small salient set, judged by the activations they see, deserves protection. Guard them, and shrink the rest. Quick check now.

One question is coming up. Let's see if it clicked.

How does scaling weights before we shrink them work? Let's try to understand. In a simple way. Last time we found the salient weights, the slice that carries the model.

The easy fix, keep them in full precision. But two formats slow the chip. Here is a gentler trick. Rounding leaves a leftover, a fixed fraction of the grid step.

That step hurts a small weight more than a big one. So take an important weight and multiply it up before rounding, say by four. Now it sits higher, and the step is a smaller slice of it. But scaling a weight changes the answer.

Multiply it by four, and its output is four times too big. We cannot inflate weights for free. A weight times its input stays equal if you scale the weight by four and divide the input by four. The product never moves.

So push the scale onto the matching input channel. The weight goes up, its activation comes down the same amount. The output holds, the weight rounds cleanly. Each input channel gets its own scale.

Loud channels get a big boost, quiet ones stay near one. A search picks the scale that shrinks error. And notice what we skipped. No retraining, no gradients.

Just multiply the weights, divide the inputs, done once. One uniform format, and the chip stays happy. So, before shrinking, scale the important weights up and their inputs down. The answer is unchanged, yet the weights that matter round with far less error.

Quick check now. One question is coming up. Let's see if it clicked.

How does choosing between speed and precision work? Let's try to understand. In a simple way. Say you shrink a model from sixteen bits per weight down to four.

It is not just smaller on disk. It runs faster too. To write each new word, the model must read every single weight from memory. For a big model, that is billions of numbers, moved for one word.

So the slow part is not the math. It is moving all those weights across the memory pipe. The chip mostly sits and waits for numbers to arrive. Now shrink each weight.

Four bits instead of sixteen means one quarter of the bytes to move. The weights arrive sooner, so the next word comes out sooner. But there is a price. Fewer bits means a coarser grid of allowed values.

Every weight is rounded harder, so tiny errors creep into every number. And the damage is not steady. From sixteen bits down to about four, quality barely moves. Push below four, and accuracy falls off a cliff.

So there is a sweet spot. Four bits buys most of the speed, for almost none of the accuracy. That is why most shipped models stop there. And the right point depends on the job.

A quick chat assistant leans toward speed. A careful math or code model leans toward precision. You pick where to sit. So, fewer bits move less data, so the model runs faster, but rounds harder.

Four bits is the usual sweet spot between speed and precision. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is GGUF and NF4? Let's try to understand. In a simple way. When you download a quantized model, two names keep appearing.

GGUF, and NF4. Here is the trick that clears it up. NF4 is a number type. GGUF is a file format.

Take NF4 first. It sets how a single weight is stored, squeezed into four bits. So it is the shape of one number, and nothing more. Its clever part is where the sixteen levels sit.

Weights bunch up near zero in a bell curve. So NF4 crowds more levels near zero, where the weights really are. Normal curve, and float, four bits. Now GGUF.

It does not shrink numbers at all. It is a file format. One neat box that holds a whole quantized model. Inside that single file sits everything.

The quantized weights, the settings, even the tokenizer. Open it, and a tool has the full model, ready to run. And GGUF was built for plain hardware. Laptops, and everyday processors.

A tool like llama.cpp reads the file and runs the model, with no special setup. GGUF even carries its own family of four bit recipes inside. So the two names never compete. One says how a single number looks.

The other says how the whole model is boxed up. So, NF4 is a smart four bit number type, shaped to the bell curve. GGUF is a file format that packs a whole model into one laptop ready file. A number, and a box.

Quick check now. One question is coming up. Let's see if it clicked.

How does the trade-off between size and smarts work? Let's try to understand. In a simple way. A model is a big pile of weights.

Its size is the number of weights, times the bits used to store each one. Sixteen bits, or four. Why shrink? To make it fit.

Your graphics card, or your laptop, has a fixed pile of memory. A full size model can be too big to load. But smarts means accuracy. How good the answers are.

Fewer bits round every weight harder, so some accuracy can quietly slip away. Here is the tension. Squeeze it smaller, and it fits on cheaper hardware. Squeeze too hard, and it gets dumber.

Size on one side, smarts on the other. And the loss is not steady. From sixteen bits down to about four, accuracy barely moves. Shrink past four, and it starts to fall away fast.

Here is the twist. For the same memory budget, a large model squeezed to four bits often beats a small model kept full size. More smarts per gigabyte. So four bits is the usual sweet spot.

You keep almost all the smarts, while cutting size to a quarter. Most shipped models land there. How far you go depends on your box. A tiny device leans hard on size.

A roomy server can keep more bits, and more smarts. So, fewer bits shrink the model to fit, but cost a little smarts. Four bits is the sweet spot, and a big model squeezed small beats a small one kept large. Quick check now.

One question is coming up. Let's see if it clicked.

How does running huge models on your computer work? Let's try to understand. In a simple way. To run a model, every weight must sit in fast memory.

Your graphics card holds only a small, fixed pool. That pool is the wall. How big is that model? A large one has seventy billion weights.

At sixteen bits each, that is a hundred and forty gigabytes. But a gaming card holds maybe twenty four gigabytes. A hundred and forty does not come close. The door is shut.

So shrink each weight to four bits, half a byte. The model drops to about thirty five gigabytes. One quarter the size. Thirty five still tops twenty four.

So split the model. Put as many layers as fit onto the graphics card. The rest run on your processor. Now the model lives in two homes.

The fast card runs the layers it can hold. The processor takes the overflow. One more trick. The file can be memory mapped.

Pieces stream from your disk only when needed, so you never hold the whole thing at once. You do not wire this yourself. Tools like Ollama and llama.cpp read one quantized file and spread it across card, memory, and disk. The payoff.

A model that once needed a rack of datacenter chips now runs on the laptop on your desk. So, four bits makes the weights fit, and splitting spreads what is left across card, processor, and disk. That is how a huge model runs at home. Quick check now.

One question is coming up. Let's see if it clicked.