Inference engineering

How does token by token generation work? Let's try to understand. In a simple way. When you ask a model a question, the answer looks like one finished block of text.

It is not. The model writes one small piece at a time. Each piece is called a token. Roughly a word, or a chunk of a word.

In one step the model produces exactly one of them. Then the new token is glued onto the end, and the whole thing goes back in. The model reads it again, and guesses the next one. Here is the part that matters.

Every single token costs one full pass through every layer of the model. Billions of parameters, for one word. What comes out of that pass is a score for every token in the vocabulary. Tens of thousands of scores.

The model picks one. Your prompt is different though. All of its tokens can be read in one go, in parallel. That first pass is called prefill.

After that comes decode. One token, one pass, over and over. Prefill is a sprint. Decode is a slow drip.

Why not write ten tokens at once? Because token ten depends on token nine. The model cannot pick it until token nine exists. The loop keeps running until the model produces a special stop token.

That is the model saying, I am finished. So, prefill reads your whole prompt at once, then decode writes one token per pass, until stop. Every hard problem in serving starts there. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the difference between latency and throughput? Let's try to understand. In a simple way. Ask whether a serving system is fast and you get two different answers.

One is about your wait. The other is about the whole machine. Latency is your wait. Start a stopwatch when you press send, stop it when the answer finishes.

One request, measured in seconds. For a chatbot the number that stings is time to first token. How long the screen stays empty before the first word appears. Then comes time per output token.

The gap between one word and the next. Small gaps, and the text streams faster than you read. Throughput is a rate, not a wait. Total tokens per second across everyone on the server.

Latency is a duration. Throughput is work per second. Think of a highway. Latency is how long your car takes to reach the exit.

Throughput is how many cars pass that exit each hour. And the two fight. Group many requests onto the graphics card and total work per second climbs. But your request waits its turn, so your own wait grows.

Averages also lie. Teams quote the ninety fifth percentile, the wait that only one request in twenty is worse than. So fast means nothing on its own. A chat box is judged on latency.

An overnight job across a million documents is judged on throughput. So, latency is one request's wait. Throughput is the server's rate. Two clocks, and pushing one usually costs the other.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the VRAM bottleneck in model serving? Let's try to understand. In a simple way. A graphics card ships with a fixed pool of fast memory, called VRAM.

A big data center card holds eighty gigabytes, and not one byte more. Before any user arrives, the weights are copied in, and they stay there for the server's whole life. Count it yourself. Parameters times bytes per parameter.

Seven billion parameters at two bytes each is fourteen gigabytes, gone before anyone types. So the number that matters is not the eighty. It is what is left after the weights. That leftover is the only room your users get.

Every conversation in flight parks its own scratch memory in that leftover, and it grows with every token. Ten users, ten slices. So memory, not math, sets how many people you can serve at once. When the leftover runs dry, the next request waits outside.

There is a second squeeze. To write one token, the card must read every weight out of memory. All fourteen gigabytes, for one token. So the math units finish early and sit waiting for bytes.

That is memory bound. The limit is bytes moved, not math done. And a seventy billion parameter model needs a hundred and forty gigabytes. That fits on no single card, so it is split across several.

So, VRAM holds the weights first, rents what is left to your users, and is read again for every token. Memory is the bottleneck, not the math. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is the KV cache? Let's try to understand. In a simple way. To write the next word, the model looks back at every word before it.

From scratch each time, that work repeats over and over. Inside attention, every word becomes two things. A key, which says what it offers, and a value, which is what it hands over. Now the useful part.

Once a word is in the past, its key and value never change. Recomputing them is pure waste. So the server keeps them. Each key and value saved in memory the first time it is computed.

That store is the KV cache. Each step gets cheap. Compute a key and value for the one new word, append them, and read all the older ones from the cache. The gap is huge.

Without a cache, step one thousand redoes a thousand words. With it, that step does one word of fresh work. The cache is not free though. It is memory.

Two vectors per word, in every layer, for every attention head. And it only grows. Every token adds a slice, so a long chat quietly eats gigabytes, and every user needs a cache of their own. One thing it is not.

This is not the model remembering you. The cache lives for one conversation, then it is thrown away. So, the KV cache saves each word's key and value once, and every later step reuses them. Speed, bought with memory.

Quick check now. One question is coming up. Let's see if it clicked.

How does continuous batching work? Let's try to understand. In a simple way. Serving one request alone wastes the card.

A single token still reads every weight out of memory, and the math units sit half idle. So run many requests in one step. The weights get read once and shared by the whole batch. Eight users cost little more than one.

The old way was static batching. Collect a group of requests, launch them together, and run that fixed group until every one is finished. But answers are not the same length. One reply stops after twenty tokens, another runs to five hundred.

Finished seats then sit there doing nothing. And nobody new can join. A request arriving one step late waits for the whole batch to drain, even with seats sitting empty. Continuous batching changes when that decision is made.

The scheduler picks the batch before every token step, not once per group. That is iteration level scheduling. So the moment a reply hits its stop token it leaves, and a waiting request takes that seat on the very next step. Nothing about the model changed.

Same card, same weights. But the seats stay full, so throughput climbs several times over, and the queue drains faster. It is not free though. Every seat needs its own memory, so spare memory caps the batch.

A fuller batch also slows each person's tokens. So, continuous batching schedules every step instead of every batch. Finished replies leave, waiting ones join at once, and the card never idles. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is PagedAttention? Let's try to understand. In a simple way. The keys and values for one chat have to live somewhere on the graphics card.

The simple way is one long unbroken stretch per user. But nobody knows how long the answer runs. So the server books room for the longest reply allowed. Two thousand tokens, for every request.

Then the reply stops after eighty tokens. The rest of that stretch stays booked and untouchable, holding nothing at all. Requests finish at different times, so free memory ends up in scattered gaps. A new user needs one long stretch, and cannot fit.

PagedAttention drops the long stretch. It cuts the cache into small fixed size blocks, sixteen tokens each, and those blocks sit anywhere. Since the blocks are scattered, every sequence gets a block table. It records, in order, which physical block holds which chunk of the chat.

Nothing is reserved ahead now. A fresh block is handed out only when the last one fills. Waste shrinks to part of one block. Blocks can also be shared.

Two requests that begin with the same system prompt point at the same blocks, so it is stored once. This is an old trick. Your operating system pages memory the same way. The program sees one clean run, while the pieces lie scattered.

So, PagedAttention keeps the cache in small blocks, tracked by a table. Less waste means more users on the same card. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is number quantization? Let's try to understand. In a simple way. A model is just its weights.

Billions of learned numbers. By default each one is stored in sixteen bits, two bytes of memory. Sixteen bits can spell out sixty five thousand values. But one weight is a small number near zero.

Most of that precision is never used. So quantization picks a short list of levels. Four bits gives sixteen of them. Every weight is rounded to the nearest level and stored as that index.

An index alone means nothing. So each small block of weights, say sixty four, keeps one scale factor. Index times scale rebuilds the number. The arithmetic.

Sixteen bits down to four is four times smaller. A seven billion weight model drops from fourteen gigabytes to about four. And it runs faster. Writing one token means reading every weight off the memory.

Fewer bytes to fetch, less waiting, more tokens per second. Nothing is free. Rounding moves every weight a little, and those errors stack up through the layers. Quality slips, usually by a hair.

The danger is outliers. One huge weight stretches its block's scale, and its neighbors collapse onto the same level. Good schemes keep those few at full precision. Weights are the easy part.

They sit still, so you quantize once, after training. Activations change with every prompt, so most servers leave them alone. So, quantization stores each weight in fewer bits, with a scale per block. Smaller model, faster tokens, a small price in accuracy.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is model compilation? Let's try to understand. In a simple way. By default a model runs one operation at a time.

Multiply, then add, then an activation. Python hands each one to the card, and waits. And each step reads its input out of memory, then writes its answer back. The math is quick.

The trips are what cost you. A compiler watches one run and writes down what happened. Out comes a graph. Every operation, in order, with no Python in the middle.

Seeing the whole graph, it can fuse. Multiply, add and activation become one kernel. The values stay in registers, so one read replaces six. It also picks the code.

For your exact shapes it times several versions of the same matrix multiply and keeps whichever one wins. Launch overhead is the other tax. Each job handed to the card costs microseconds, and one decode step hands over thousands. Recorded once, they replay as one launch.

Notice what did not change. Same weights, same answers. Quantization buys speed by shrinking the numbers. Compiling touches none of them.

The bill comes first though. Compiling takes seconds, sometimes minutes, so the first request crawls. Servers pay it at startup, before any user arrives. A graph also fits one set of shapes.

Hand it a batch size it has never seen, and it compiles again. So servers pad to fixed sizes. So, compiling records the model as a graph, fuses the small steps, and replays them in one launch. Same answers, fewer trips.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is speculative decoding? Let's try to understand. In a simple way. A large model writes one token per pass.

Every pass drags all of its weights through memory. Easy word or hard, you pay the same. But most of those tokens are not hard. A comma, the rest of a common phrase.

A much smaller model would guess them right too. So put a small model in front. Same vocabulary, maybe twenty times cheaper to run. Let it race ahead and draft the next four tokens.

Then the large model reads that draft. Not to write from scratch, but to check it. For each token, would I have said this? Everything it agrees with is kept.

At the first disagreement the draft is cut, and the large model writes that token itself. And here is the surprise. What comes out is exactly what the large model alone would have written. Nothing is approximated.

The small one only proposes. The whole win is the acceptance rate. Accept three of four drafted tokens, and one expensive pass delivered four. Reject early, and the draft was wasted.

How small should the helper be? Too small, it guesses badly and its work is thrown away. Too large, and drafting costs what it saves. One catch.

This buys speed with spare capacity. On a server already packed with users, there is none spare, and the gain shrinks. So, a small model drafts, the large one checks and corrects. Same words as before, in fewer expensive passes.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is multi token prediction? Let's try to understand. In a simple way. A normal model ends in one output head.

It reads the final hidden state, writes one token, and the pass is over. So bolt on extra heads. The first predicts the next token. The second predicts the one after that.

The third goes one further. Now a single pass comes out with four tokens instead of one. The first is the real prediction. The rest are guesses.

Why does that help? Because reading is parallel and writing is not. Feeding four tokens back in takes one pass. Writing four takes four.

So the guesses go back in together. In that one pass the model asks at every spot, is this what I would have written? The further ahead a head looks, the weaker it gets. The next token is easy.

Four steps out, the guess misses more often than it lands. So do not guess one path. Each head offers a few candidates, and they form a small tree. One pass scores it, and the longest surviving branch wins.

Notice what is missing. No second model to load and serve. The heads are a few percent bolted onto the model you already have. The price is real.

Those heads must be trained, and a bigger tree means more work per pass. Push it too far and the pass slows down. So, extra heads guess several tokens at once, and one pass checks them all. Only what the model signs off on survives.

Quick check now. One question is coming up. Let's see if it clicked.

How does token streaming work? Let's try to understand. In a simple way. Without streaming, the server writes the whole answer, then sends it.

You watch a blank screen for twenty seconds. So the connection opens once and stays open for the whole answer, with every token pushed down as a tiny event. That is server sent events. Nothing got faster.

The last word still lands at the same second. Only the first word moved, and that is what people feel. Now the hard part. A token carries bytes, not letters.

An emoji can split across two tokens, so the server holds bytes until a full character exists. And once a token leaves, it is on the screen. Any check that needs the finished answer has already missed its chance. Stop words are worse.

One may arrive in three pieces, so the server trails a token behind, holding pieces back in case they spell it. Errors get strange too. The first token already sent a success code, so a later failure has to ride inside a stream that is technically fine. The user closes the tab, and nothing tells the model.

Unless the server notices the dropped connection and cancels, it writes for nobody. Tokens also do not arrive evenly. They come in bursts, then pause. That smooth typing is often the client releasing them at a steady pace.

So, streaming is one open connection carrying tokens as they are written. It buys the first word early, and costs you the right to change your mind. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a latency budget? Let's try to understand. In a simple way. Every serving team says the model should feel fast.

But fast is an opinion. A budget turns it into a number you meet or miss. So you write the number down. First word inside three hundred milliseconds.

Then under fifty milliseconds between words. Two promises, in plain numbers. The numbers are not invented. A reply that begins inside a quarter of a second feels instant.

Words faster than you read are fast enough. And a budget is spent in pieces. Time in the queue. Time reading your prompt.

Then time per token, for every token out. So you subtract. Three hundred to spend. Ninety lost in the queue, a hundred and thirty reading the prompt.

Eighty left. Anything costing more is out. A budget also needs a percentile attached. Say the ninety ninth.

Then the one slow request in a hundred is what you design for. Watch what the number buys. Bigger batches lift total output and stretch every wait. So you grow the batch until the budget nearly breaks, and stop.

That beats plain throughput. Goodput. Requests per second finishing inside the budget. A server answering twice as many, all of them late, scores nothing.

No single number fits every job. A voice agent needs the first word in a heartbeat. A nightly document sweep can wait a minute. So, a latency budget is a number, with a percentile, split across the parts of the wait.

It turns tuning arguments into arithmetic. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a prompt traffic jam? Let's try to understand. In a simple way. Requests do not arrive politely spaced out.

They come in clumps. A class starts, a batch job fires, and ten long prompts land in one second. Reading a prompt is the expensive part. Before one word comes back, the whole prompt goes through the model in one burst of math.

That is prefill. The card does one thing at a time. So while it chews those prompts, every chat already mid answer gets nothing. Their words stop.

That is the jam. So nothing runs the moment it arrives. Requests land in a waiting queue, and a scheduler picks who moves at every step. The main fix is chunked prefill.

Cut the long prompt into slices of a few hundred tokens. Each step runs one slice, plus everyone else's next token. Nobody freezes. Order matters too.

First come first served is fair, but a giant prompt at the head holds up ten small ones. Shortest first clears the line, and starves the giant. And if memory runs out, something gives. The scheduler evicts a running request, drops its cache, and puts it back in the queue.

Work repeated, nothing crashed. Sometimes the honest answer is no. Once the queue is longer than the promise allows, taking more work helps nobody. The server refuses fast, with a busy code.

So, a traffic jam is prompt reading blocking everyone's tokens. Slice the prompts, order the line, refuse what you cannot serve. Quick check now. One question is coming up.

Let's see if it clicked.

How does serving thousands of models with LoRA adapters work? Let's try to understand. In a simple way. Every customer wants their own fine tuned model.

But one full copy of a seven billion model is fourteen gigabytes. A thousand copies will never fit. A LoRA fine tune is not a new model. It is a small patch.

Two thin matrices beside each big weight matrix. Megabytes, not gigabytes. So the server holds one base model, and thousands of tiny adapters beside it. Fourteen gigabytes once, plus a few megabytes per customer.

You could add a patch into the weights. That is merging, and it is fast. But then the card serves one customer. So keep them separate.

Now the hard part. One batch holds requests from different customers, so different adapters. Grouping by adapter gives many small batches, and the card sits idle. The fix is a batched adapter kernel.

The big base multiply runs once for the whole batch. A tiny second multiply then gives each row its own adapter. Adapters live in a pool. Busy ones sit on the card, quiet ones wait in host memory.

Pulling one back takes milliseconds, not a model load. It is not free. The extra thin multiply costs a little time on every token, and a higher rank costs more. A small dent, not a cliff.

So, one base model, many small adapters, one mixed batch. That is how a single card serves thousands of custom models at once. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is the tradeoff between speed, cost, and quality? Let's try to understand. In a simple way. Every serving team wants three things.

Answers that arrive fast. Tokens that cost very little. And answers that are actually good. But all three are bought with the same thing.

Seconds on a graphics card. And a card does a fixed amount of math per second. Turn the first knob. Pack more requests into one batch.

The cost per token drops, because the card is shared. Every single user waits longer. Second knob. Serve a smaller model.

Fewer numbers to move, so tokens come out faster and cheaper. And the hard questions start getting missed. Third knob. Let the model think before it answers.

Every thinking token is real compute, so quality climbs while speed and cost both get worse. Notice the shape. Fast and cheap means a small model in big batches. Fast and good means a big model, running almost alone.

You pick two. Those knobs only slide you along one curve. Better kernels and better hardware move the whole curve outward. That is the only real win.

And watch the quality side. A cheaper setup often looks fine on average. It fails on the rare hard question. So measure the tail.

So the real job is choosing. Name the one axis you refuse to trade, then spend the other two to buy it. So, speed, cost, and quality all pull on the same graphics card. Every knob pays for one of them with another.

Quick check now. One question is coming up. Let's see if it clicked.