Small language models

What exactly is a small language model? Let's try to understand. In a simple way. A model's size is its parameter count.

Giants have hundreds of billions of learned weights. A small model has maybe one to ten billion. Same design, just far fewer knobs. Here is the thing.

To produce each token, the model touches basically all of its weights. So inference cost scales with parameter count. Ten times smaller means roughly ten times cheaper and faster. Now look at real workloads.

Sorting support tickets. Summarizing emails. Pulling a date from a form. Most of what companies run is routine.

A frontier giant is overkill there. It's like using a sledgehammer to hang a picture. It works. But a small hammer does the job just as well, and you can afford one in every room.

And small models got seriously good. Train a small model on trillions of tokens, far past the old compute optimal recipe, and it punches way above its size. Researchers call this over-training. Speed matters too.

Each token means reading every weight from memory. Fewer weights, less to read, so replies stream back fast. That's the difference between a chat and a wait. And there is control.

A small model runs on hardware you own. Your data stays with you. No API bill, no rate limits, no vendor deciding your fate. So here is the gist.

Inference cost scales with size. Most tasks are routine. And well fed small models are now good enough. Cheap, fast, private, and yours.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a phone-sized model? Let's try to understand. In a simple way. Here is the thing.

A model is just its parameters. Millions or billions of stored numbers, called weights. To run it, every one of them has to sit in the phone's memory. So the math is simple.

Size equals parameter count times bytes per parameter. A seven billion parameter model, at two bytes each, needs about fourteen gigabytes. Now look at the phone. A flagship has maybe eight gigabytes of RAM.

And the operating system, the camera, your apps, they all want their share. The model gets a slice. So fourteen gigabytes will not fit. At that precision, the comfortable zone is around one billion parameters, maybe two.

That is the real budget. And weights are not the whole bill. The conversation costs memory too. The model keeps a KV cache, and it grows with every token of context you feed in.

Speed has its own wall. For each new token, the chip reads nearly all the weights from memory. So memory bandwidth, not raw compute, sets your tokens per second. One more limit.

Phones throttle when they get hot. Push a big model hard and the chip slows down, and the battery drains fast. So, what fits is set by three walls. Memory for the weights and the cache, bandwidth for speed, and heat for how long you can keep going.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is quantization? Let's try to understand. In a simple way. A language model is basically a huge list of weights.

Just numbers. Each one usually stored in sixteen bits, a format called half precision. Here is the idea. Keep the weights, but store each one with fewer bits.

Eight bits. Or even four. That is quantization. Same model, shrunken numbers.

Think of rounding prices. Two point seven one eight becomes two point seven. You lose a little detail, but the value is still close enough to use. How does it work?

Take a group of weights, find their range, and squeeze it onto a small grid of integers with one shared scale factor. Then round. Now the payoff. A seven billion parameter model at sixteen bits is about fourteen gigabytes.

At four bits, three and a half. Four times smaller, and faster, because memory is the bottleneck. Why does the model survive this? Because neural networks are redundant.

Millions of weights vote together, so a tiny rounding error on each one mostly cancels out. But it is not free. A few weights are outliers, way bigger than the rest. Squash them carelessly and quality drops.

Good methods handle those with extra care. Two flavors you will hear. Post training quantization compresses a finished model. Quantization aware training bakes the rounding in while training, so the model learns to tolerate it.

So, quantization stores the same weights in fewer bits, using a scale and a round. Less memory, more speed, a small hit to quality. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is distillation? Let's try to understand. In a simple way. Here is the setup.

You have a big, expensive model that's really good. Call it the teacher. And you have a small model you want to train. The student.

Now, the obvious plan is to train the student on the teacher's final answers. But that throws away almost everything the teacher knows. Because the teacher does not just say, the answer is cat. It says, ninety percent cat, nine percent dog, a tiny bit of car.

That full spread is called a soft target. Those small probabilities encode how concepts relate. Dogs look a bit like cats. Cars do not.

Hinton called this dark knowledge, and it's the real treasure. So in distillation, the student is trained to match the teacher's whole probability distribution, not just the top answer. The loss compares the two distributions, usually with KL divergence. One trick makes this work.

Temperature. You soften the teacher's outputs, so those tiny probabilities get louder, and the student can actually hear them. For language models, the teacher can also just generate training text. The student learns from millions of teacher-written examples.

That's how many small open models got so good. The payoff is huge. A student a tenth the size can keep most of the teacher's skill, because it skipped the hard part. Someone smarter already mapped the terrain.

So, distillation is a big model teaching a small one, by sharing its full beliefs, not just its answers. Small model, borrowed wisdom. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is LoRA? Let's try to understand. In a simple way. Say you have a small model, and you want it to learn your domain.

The classic way is fine-tuning. You update every single weight. That's billions of numbers, and it needs serious GPU memory. LoRA takes a different route.

It freezes the whole base model. Not one original weight changes. Instead, it adds tiny trainable pieces on the side, called adapters. Here is the trick.

Each big weight matrix gets a bypass: two thin matrices, A and B. Multiply them, and you get a correction to add on top. Thin is the key word. The pair shares a small inner size, called the rank.

That's the low rank in LoRA: low rank adaptation. Do the math on one layer. A thousand by a thousand matrix holds a million weights. Two rank eight strips hold just sixteen thousand.

That's over sixty times smaller. During training, gradients flow only into A and B. Often that's under one percent of the parameters. So you can specialize a model on a single everyday GPU.

And at inference? You can merge the adapter straight into the base weights. One clean matrix again. Zero extra latency.

Or keep adapters separate. One base model, plus a legal adapter, a medical adapter, a support adapter. Each is a few megabytes. Swap them like memory cards.

So, LoRA freezes the base model and trains two thin matrices per layer. Full fine-tuning quality, at a tiny fraction of the cost. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a domain-specific small model? Let's try to understand. In a simple way. A big general model is a jack of all trades.

Poetry, tax law, Python, cooking. But your product probably needs just one narrow slice. So flip the strategy. Take a small model, and spend every one of its parameters on one field.

Medicine. Law. Your company's own codebase. That is a domain-specific small model.

Picture a two billion parameter model raised on radiology reports. It has never read a movie review. But it knows what a ground-glass opacity is, cold. Why does this work?

Language inside a domain is narrow. Fewer topics, repeated patterns, a compact style. A small model has enough capacity to master a narrow distribution. The standard recipe is called domain-adaptive pretraining.

You keep training the model on in-domain text, millions of clinical notes or contracts, so its weights shift toward that world. Then comes supervised fine-tuning on domain tasks. Question answering over contracts, report summarization. Now it doesn't just speak the dialect, it does the job.

And the results are real. On medical exams and finance benchmarks, tuned small models routinely beat general models ten times their size. Inside their lane. The catch.

Outside the domain, they get brittle. Ask the radiology model for travel tips and quality falls off a cliff. Specialization is a trade, not a free lunch. So, the gist.

Narrow the world, and a small model can be the expert in the room. Cheaper, faster, and often more accurate where it counts. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is running AI at the edge? Let's try to understand. In a simple way. When you ask an AI something, your words usually travel to a data center.

A big model answers there, and the reply travels back. Edge AI flips this. The model runs right on the device itself. Your phone, your car, your watch.

No server involved. Why bother? First, it works with no internet. Think of a translator app on a plane, or a car assistant deep inside a tunnel.

Second, privacy. Your voice, your photos, your messages never leave the device. There is nothing to send, so there is nothing to leak. Third, latency.

No network hop means no waiting. The reply can start in milliseconds, which is why on device keyboard suggestions feel instant. So how does it run? A small model file ships inside the app, and an inference engine executes it locally on the phone's own chips.

Modern phones even carry an NPU, a neural processing unit. It is built for the matrix math models need, so it sips battery instead of draining it. Here is the catch. Generation speed at the edge is mostly limited by memory bandwidth.

Every new token means reading all the weights again. Smaller model, faster stream. In practice, many apps go hybrid. Easy requests run on device, and only the hard ones escalate to a big cloud model.

So, edge AI means the model lives where the data lives. Offline, private, and instant, in exchange for a smaller brain. Quick check now. One question is coming up.

Let's see if it clicked.

How does choosing between small and large models work? Let's try to understand. In a simple way. Here is the thing.

Bigger is not better. The real question is, what is the smallest model that clears the bar on your task? That bar is called the capability threshold. So judge every job on three axes.

Quality, how good must the answer be. Latency, how fast must it arrive. And cost, how many calls per day. Now look at the task itself.

Narrow and repetitive, like tagging support tickets? A small model usually clears the bar. Open ended reasoning across many domains? That needs the large one.

Volume changes everything. One million calls a day at a tenth of the price? Small wins big. Ten expert queries a day?

Just pay for the large model. So how do you decide for real? Build a small eval set. Say a hundred examples from your actual task.

Run both models on it. Compare accuracy, not vibes. And you don't have to pick just one. In a model cascade, the small model answers first.

If its confidence is low, the query escalates to the large model. A router does the same job upfront. A tiny classifier reads each query and sends easy ones to the small model, hard ones to the large one. Most traffic is easy.

So, the rule. Define the bar. Measure both models on your own data. Pick the smallest one that clears it.

And when in doubt, cascade. Quick check now. One question is coming up. Let's see if it clicked.