AI infrastructure, from GPUs up

Why do AI models run on gaming chips? Let's try to understand. In a simple way. An AI model, deep down, does one kind of math.

It multiplies huge grids of numbers together, again and again. Billions of tiny multiply and add steps. Here is the key. Most of those little steps do not depend on each other.

So they can all happen at the very same time. A normal computer chip, the CPU, has just a few very fast cores. It is brilliant at doing one hard thing after another. A gaming chip, the GPU, is the opposite.

Thousands of small, simpler cores, all running the same step side by side. Why was it built like that? For games. A screen holds millions of pixels, and each one needs the same little bit of shading math.

And here is the twist. Shading a million pixels and running a neural network are the same shape of work. Piles of identical, independent arithmetic. So a GPU does not run each step faster than a CPU.

It runs thousands of them at once. For this kind of work, that wins by a mile. This started as a lucky match. Games needed this power, and AI turned out to need the very same thing.

Today the biggest of these chips are bought mostly to run AI. So AI runs on gaming chips for one reason. The math is a huge pile of the same simple step, and the GPU was already built to do many at once. Quick check now.

One question is coming up. Let's see if it clicked.

How do thousands of cores live on one chip? Let's try to understand. In a simple way. A modern gaming chip is a single slab of silicon, smaller than a postage stamp.

Packed onto it are thousands of cores. Sometimes more than ten thousand. The first trick is that each core is tiny. It does just one humble job.

Multiply two numbers, then add. Something that simple takes very little space. But even tiny workers cannot just be dumped in a heap. Ten thousand of them would be chaos.

They need to be organized. So the cores are grouped into small teams. Each team is a cluster of cores that sit together. On these chips a cluster is called a streaming multiprocessor.

Inside one cluster live dozens of these cores. Plus a few shared parts. A scheduler to hand out work, and a patch of fast scratch memory. Now the real trick.

The cores in a team do not each think for themselves. One instruction is fetched, and the whole team runs it. The same step, on different numbers. This is why a core stays so small.

It has no control room of its own. The thinking is shared, so a core is almost pure arithmetic. And the whole chip is one team design, copied over and over. Engineers lay out a single cluster, then stamp that tile across the silicon.

So thousands of cores live on one chip like this. Tiny cores, gathered into teams that share one instruction, and that team copied again and again. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is the high bandwidth memory a GPU needs? Let's try to understand. In a simple way. A GPU is packed with thousands of tiny cores.

Each does math at a blistering speed. But a core cannot invent numbers. It must be fed them, from memory. Here is the catch.

Very often, the slow part is not the math. It is fetching data from memory to the cores. People call this the memory wall. Memory has two sides.

How much it can hold, and how fast it can pour data out. That pour rate is the bandwidth. For a GPU, bandwidth is what matters most. Ordinary memory, the kind of stick inside a laptop, sits far across the board.

It talks through a narrow channel. Too slow to keep thousands of cores busy. So the fix has a name. High bandwidth memory.

Instead of one distant stick, several memory chips are stacked into a tiny tower, right beside the GPU. Because it sits so close, it can be wired with an enormous road. Thousands of connections at once, not one narrow lane. Data travels side by side every tick.

The payoff is speed. This memory can pour several terabytes of data every single second. Fast enough that the hungry cores rarely sit waiting. And here is why it matters.

To write each word, the model must read all of its weights from memory. Bandwidth sets that speed limit. Faster memory, faster answers. So, high bandwidth memory is stacked, close, and wired wide.

That is how it feeds the cores at terabytes per second. Quick check now. One question is coming up. Let's see if it clicked.

How do multiple GPUs talk to each other? Let's try to understand. In a simple way. A modern AI model is enormous.

Far too big to fit on one GPU. So we split it across many GPUs, each holding one slice. But those slices are not independent. To finish a step, a GPU needs results sitting on its neighbors.

So the chips trade numbers, constantly. The simple way to pass them is through the CPU, over a slot called PCIe. But routing every message through that middleman is slow. The fix is to let the GPUs talk directly.

A cable runs from one GPU into the next, reading its neighbor's memory with no CPU involved. These links are called NVLink. And these links are wide. A normal PCIe slot moves tens of gigabytes a second.

An NVLink moves hundreds. Many times the room for data. With eight GPUs in one box, you do not want a chain. Instead a switch wires every GPU to every other, all at full speed.

Nvidia's is the NVSwitch. The move they repeat most is an all reduce. Each GPU holds one piece. The links add every piece, then hand the same total back to all of them.

Here is why the link matters. The GPUs must sync constantly. If the connection is slow, fast chips finish, then wait idle. The interconnect is as vital as the cores.

So multiple GPUs talk over direct links, not the CPU. Wide NVLink cables and a switch let every chip share numbers at full speed. Quick check now. One question is coming up.

Let's see if it clicked.

What makes an AI server so heavy? Let's try to understand. In a simple way. An AI server is shockingly heavy.

One box can weigh as much as an adult. A full rack, more than a car. So where does the weight come from? Here is the surprise.

It is not the chips. The GPUs themselves are light, about as heavy as a book. The weight is everything wrapped around them. Start with the real cause.

Heat. Eight or more GPUs together burn thousands of watts, like a closet full of space heaters. Every watt becomes heat. To pull that heat off each chip, it wears a thick block of metal, a heat sink.

Mostly copper and aluminum. Copper is dense, and dense means heavy. The biggest machines give up on air. They run liquid cooling instead.

Cold plates on every chip, plus pipes, pumps, and coolant. That metal plumbing adds real mass. Then there is power. Feeding thousands of watts needs thick copper bars, called busbars, and rows of heavy power supplies.

More copper, more iron, more weight. And it all must be held rigid. A thick steel and aluminum frame stops the parts from sagging or shaking. That structure is heavy on purpose.

Now stack many servers into one tall rack, wire them together, and cool the whole cabinet. A loaded rack can pass a metric ton. So the weight is not the silicon. It is the cooling, the power delivery, and the metal frame.

You are really lifting copper, plumbing, and steel. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the special network connecting AI servers? Let's try to understand. In a simple way. One server holds only about eight GPUs.

But a large training run needs thousands. So the job spreads across hundreds of servers. Inside one box, the eight chips already talk over fast NVLink cables. But that only reaches inside the box.

The boxes still need to reach each other. And they cannot work alone. Every training step, all of them add their results and share the total. The same numbers cross every server, again and again.

The ordinary office network that carries email and web pages is too slow and jittery for this. One stall would freeze thousands of chips. So an AI cluster adds a second network, used only for GPU traffic. Engineers call it the back end fabric.

Its trick is called RDMA. A GPU in one server writes straight into a GPU inside another, skipping both CPUs and the operating system. These links are extreme. Each one moves about four hundred gigabits a second, with a tiny, very steady delay.

The servers plug into switches arranged in a tree, so any box can reach any other at full speed. No chip sits in a slow lane. This fabric is the backbone of the whole supercomputer. If it slows down, every GPU waits.

That is why it gets its own dedicated wiring. So thousands of servers are stitched into one machine by a dedicated high speed network. It moves numbers straight between GPUs, keeping every box in sync. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the difference between training and inference hardware? Let's try to understand. In a simple way. An AI model lives two lives.

First it learns its weights. That is training. Later it uses those frozen weights to answer, over and over. That is inference.

Training runs two passes. A forward pass to make a guess, then a backward pass to measure the error and nudge every weight. Inference runs the forward pass only. That backward pass is greedy.

To adjust a weight, you must remember every intermediate from the forward pass, plus gradients and optimizer state. Memory balloons to several times the model. Inference carries none of that. No gradients, no optimizer state.

It reads the weights, emits a word, and drops the scratch work. Far lighter on memory. So training wants the biggest chips. Maximum memory and compute, lashed together, running flat out for weeks.

One long marathon. Inference wants the opposite. Cheaper chips, often smaller, tuned to answer fast. Because this job runs billions of times, once per request.

Timing splits them too. Training is offline, so nobody waits. Inference is live. A person watches the cursor, so every millisecond of latency counts.

And here is the twist. Training is a few huge runs. Inference never stops. Over a model's life, most of the hardware bill is inference.

So, training and inference are two jobs. One learns with a heavy backward pass, the other just answers. Different needs, different hardware. Quick check now.

One question is coming up. Let's see if it clicked.

How does orchestrating a giant training job work? Let's try to understand. In a simple way. Training a frontier model is one job, but it runs on thousands of GPUs at once, all training the same model.

You do not pick those GPUs by hand. You submit the job to a scheduler, the cluster's traffic controller. It finds enough free GPUs and reserves them. But a job like this is all or nothing.

Every GPU must start together, or none can. So the scheduler waits for the whole set, then starts them at once. Now it wakes them up. One process per GPU, across many machines.

Each one gets a number, its rank, and the address of the others. From there they run in lockstep. Every step, each GPU does its share, then they sync and average before the next. The slowest one sets the pace.

Here is the hard part. At this scale, hardware breaks constantly. Thousands of chips running for weeks means something fails every day. And one dead GPU can kill the whole job.

The fix is the checkpoint. Every so often, the whole job pauses and saves its progress, the weights and the optimizer state, onto disk. So when a chip dies, you do not start over. The scheduler swaps in a healthy one, everyone reloads the last checkpoint, and the run continues.

Minutes lost, not weeks. So orchestration is three jobs. Reserve every GPU at once. Launch and connect them.

And checkpoint often, so one failure costs minutes, not the whole run. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the software stack for serving a model? Let's try to understand. In a simple way. A trained model is just a file of frozen weights on a disk.

By itself, it answers no one. Something must load it and run it. So we stack software on the hardware. Each layer does one job, then passes work upward, turning a file into a service.

At the bottom sit the kernels. Tiny programs, written for the GPU, that do the raw matrix math. Libraries like CUDA. The muscle.

Above it sits the inference engine, like vLLM. It loads the weights, holds the key value cache, and drives the forward passes that produce each word. Wrapping the engine is the model server. It opens an interface that looks like the big providers.

Requests in, answers out. The server also holds a queue. When many arrive at once, it sets the order and feeds them to the engine. No prompt gets lost.

One GPU serves only so many people. So an orchestrator, often Kubernetes, runs many copies of the stack, and a load balancer spreads the requests. When traffic surges, the orchestrator starts more copies. When it fades, it shuts them down.

The service grows and shrinks with demand. The layers stay separate on purpose. Because the interface is standard, you can swap a faster engine underneath, and the app never notices. So the serving stack is layers.

Kernels do math, an engine makes words, a server handles requests, an orchestrator scales it. Weights become a live service. Quick check now. One question is coming up.

Let's see if it clicked.

How does quantization make models faster? Let's try to understand. In a simple way. A model is a giant pile of numbers called weights.

Billions of them sit in memory, each stored using some fixed number of bits. By default, each weight uses sixteen bits. Picture sixteen tiny switches, each on or off, describing one number in fine detail. Quantization asks a bold question.

What if we describe each weight with fewer bits? Maybe eight, or even four. Fewer bits per weight, and the whole model shrinks. The same network now fits in a fraction of the memory, on cheaper chips.

But here is the real speedup. To produce each word, the chip must haul every weight out of memory. Fewer bits means fewer bytes to move, so answers arrive faster. How do you shrink a number?

You look at the spread of values, lay down a small grid of allowed levels, and snap each weight to the nearest one. Those levels are coarse, so you also keep one scale factor for the group. It is the ruler that stretches them back to real values. There is a catch.

Snapping to the grid rounds every weight a little. Push the bits too low, and that rounding starts to hurt the answers. So where is the sweet spot? Eight bits barely hurts.

Four bits is wildly popular. And the touchiest weights can stay at higher precision. So, quantization stores each weight in fewer bits. Less memory to hold, fewer bytes to move, and the same model runs faster.

Quick check now. One question is coming up. Let's see if it clicked.

How does request batching work? Let's try to understand. In a simple way. Serving requests one at a time.

To produce each word, a GPU hauls its entire pile of weights out of memory. Here is the problem. Reading those weights from memory is slow, but the math is quick. So the cores finish early and sit mostly idle.

Batching fixes this. Instead of one request, you gather many requests and push them through the model together, in a single pass. Now the real trick. You still load each weight just once.

But that one weight now does math for every request in the batch at once. So the expensive memory trip is now shared across the whole group. Total words per second climbs, though the chip works no faster. That number is throughput.

There is a price. A request may wait a moment while the batch fills up. For one user, it is no faster. The win is across everyone.

Why not batch a thousand? Each request needs its own scratch memory for the conversation so far. Fill that up, and the batch cannot grow further. Real requests arrive at odd times and finish at different lengths.

Make everyone wait for the slowest, and the chip falls idle again. So servers batch continuously. The moment one request finishes, a new one takes its seat. The batch stays packed every step.

So, batching serves many requests in one pass. Each weight loads once and serves all, so throughput soars while the hardware stays the same. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is renting versus buying AI servers? Let's try to understand. In a simple way. You need GPUs for AI.

Two ways to get them. Rent time on someone else's machines in the cloud, or buy your own. Renting is simple. You pay by the hour, nothing upfront.

The cloud provider owns the hardware, powers and cools it. When done, you hand it back. Buying is the opposite. One big payment upfront.

The machines are yours, in your own building. Keeping them running is now your job. Renting bends to your needs. Need a thousand GPUs for an afternoon?

Spin them up, then release them. You pay only for hours used. A machine you own is different. It costs the same running flat out or sitting idle.

So what matters is how busy you keep it. Here is the crossover. Keep your GPUs busy for months, and owning wins on cost per hour. Bursty or short jobs, and renting wins.

But renting is not free. Rent the same GPU around the clock for a couple years, and you pay several times its sticker price. Owning has a hidden bill. Power, cooling, floor space, a team to run it.

And the chip ages. Today's fast GPU is old within years. Renting hands those worries to the provider. The newest chips in minutes, no maintenance, no machine going obsolete in your closet.

So most teams do both. Rent to experiment and absorb spikes. Buy for the steady, always on load. Renting buys speed, owning buys cheap hours.

Quick check now. One question is coming up. Let's see if it clicked.

How does a training run's cost add up? Let's try to understand. In a simple way. The bill starts simple.

You rent GPUs by the hour. So cost is the number of chips, times the hours, times the price per chip-hour. The tricky part is the hours. How long a run takes depends on how much math it must grind through.

That math has a rule of thumb. The total work is about six, times the parameters, times the training tokens. We count it in floating point operations, flops. Each GPU crunches only so many flops per second.

That is its throughput. Line up thousands and the throughputs add up. But you never hit the sticker number. Waiting and communication eat into it.

A chip may reach forty percent of its peak. So divide the work by the real speed. That gives you seconds. Turn seconds into GPU-hours, multiply by the price per hour, and out comes a dollar figure.

Put in real numbers. A big model on trillions of tokens needs thousands of GPUs running for weeks. The bill lands in the millions of dollars. And that is just one clean run.

Add the failed attempts, the restarts, and the experiments before it. The true bill is several times larger. The big lever is simple. Cost tracks parameters times tokens.

Double the model size or the data, and you roughly double the price. So, a training run's cost is work, divided by real speed, times the price per hour. Parameters and tokens set the work. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the cost of a single API call? Let's try to understand. In a simple way. When you call an AI model, you are billed not per call, but per token, the pieces of text going in and coming out.

And there are two prices. The tokens you send in, your prompt, are cheap. The tokens the model writes back cost several times more. Why the gap?

Your whole prompt is read in one parallel pass. But the reply is written one token at a time, each waiting for the last. And every output token means one full trip through the model. Billions of parameters read from memory, just to produce the next word.

There is a catch. Each new token also re-reads every earlier one from the key value cache. So a long chat makes each new word pricier. The model's size sets the base rate.

A bigger model does more math per token, so it charges more. Underneath, the real cost is GPU seconds. The provider packs many users onto one chip at once, so they bill per token, not per second. So put it together.

A five hundred token prompt plus a five hundred token reply, at a few dollars per million tokens, costs well under a cent. So the levers are clear. Shorter prompts, a smaller model, and a cap on the output length all shrink the bill. So, one API call costs you per token.

Output tokens dominate, model size sets the rate, and long context quietly adds up. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the next generation of AI specific chips? Let's try to understand. In a simple way. The chips running AI today were built for video games.

Brilliant, but general purpose. They carry graphics circuits AI never touches. So here is the new idea. Build a chip for one job.

Throw out the extras, and hardwire what AI does most: multiply numbers and add them. At its heart is a grid of tiny multiply and add units, a systolic array. Numbers flow in from the edges, each cell handling part of the math. The trick: data keeps moving through the grid, reused at every cell, instead of running back to memory each step.

Fewer memory trips, far less energy. These chips bake the number format into the silicon. Low precision, like eight bit numbers, is built in from the start, not an afterthought. But there is a catch.

A hardwired chip is fast and efficient, but rigid. If the math shifts, a GPU adapts, a fixed chip cannot. A bet on which operations matter. Why now?

The workload finally settled. Almost everything is transformers, and transformers are mostly matrix multiply. When one operation dominates, custom silicon pays off. The next generation pushes further.

Wafer scale chips, a single dinner plate of silicon. Chips that compute inside the memory itself. Even chips that compute with light. So, an AI chip strips the extras and hardwires the math AI repeats most.

Less flexible than a GPU, but far faster and cheaper at the job that rules everything. Quick check now. One question is coming up. Let's see if it clicked.