Running AI locally

What exactly is a cloud API versus your own machine? Let's try to understand. In a simple way. There are only two places an AI model can run.

On someone else's computer, far away. Or right here, on your own. The first way is called a cloud API. The model lives on a company's servers.

You type, your words travel over the internet, their machines do the thinking, and the answer comes back. So you are really renting. You never own the model. You pay a little for each request, and you need a live connection every time.

The second way is your own machine. You download the model one time onto your computer, and it runs there. Your words never leave the device. So why choose the cloud?

The biggest, smartest models live there. Nothing to install, and the heavy work runs on their powerful hardware, not yours. And why choose your own machine? It keeps working with no internet.

After the download, each answer costs nothing. And your private data stays with you. But your own machine has a ceiling. A model runs only if it fits in your memory, so at home you often run a smaller one.

There is also control. A cloud model can change or disappear overnight. The one on your disk is yours to keep, and it answers the same every time. So, two choices.

Rent a giant in the cloud, needing internet, your data leaving. Or own a smaller model at home, running offline, your data staying put. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a model weight? Let's try to understand. In a simple way. Open up an AI model, and there is no code inside.

Just numbers. Millions, sometimes billions of them, sitting in long lists. Each single number is called a weight. It is just a value, like zero point three, or minus one point two.

Picture every weight as a tiny dial. Together, all these dials control how the model turns an input into an output. Inside the network, neurons pass signals along. A weight sits on each connection, deciding how strongly one signal pushes the next.

You will also hear the word parameter. A parameter and a weight mean the same thing: a tunable number the model learned. Where do the values come from? Training.

The model reads mountains of text and nudges every weight a little, again and again. Once training ends, the weights freeze. Running the model on your machine never changes them. It only reads these fixed numbers.

So when you download a model, you are downloading its weights. That file full of numbers is the model itself. This is why you see names like seven billion. That number counts the weights.

More weights, more room to store what the model knows. So, a weight is one learned number, a tiny dial. Parameters are the same thing. Billions of them stacked together are the whole model.

Quick check now. One question is coming up. Let's see if it clicked.

Why is a language model so large? Let's try to understand. In a simple way. Download a model to run at home, and the file is surprisingly big.

Often several gigabytes. Some reach hundreds. Far bigger than a normal app on your phone. So what is taking up all that space?

Not code. The file is packed with the model's parameters, the tuned numbers we also call weights. And there is a staggering number of them. A small model holds a few billion parameters.

A large one can hold hundreds of billions. Here is the part people miss. Every one of those numbers takes up real room on disk. A common setup stores each number in two bytes.

So the file size is really just a multiplication. The count of parameters, times the bytes each number needs. That product is how big the file is. Let's try it.

Seven billion parameters, at two bytes each, is fourteen billion bytes. That is about fourteen gigabytes, just to hold the numbers. Notice there are only two knobs here. How many parameters, and how many bytes per number.

Make either one smaller, and the whole file shrinks. That is why a model dwarfs your other files. A photo stores a few million dots. This stores billions of learned numbers, all written down.

So, a model is large for two reasons. Billions of numbers, and every number costs bytes. Count times bytes each equals the size on disk. Quick check now.

One question is coming up. Let's see if it clicked.

How does a model use your RAM? Let's try to understand. In a simple way. When you download a model, it lands on your disk.

It rests there as a big file, roomy but slow, and it cannot think while it sits. To run it, the computer copies every one of those numbers off the disk and into RAM. RAM is your working memory, the desk the processor works on. Why move it at all?

Because RAM is fast. The processor grabs a number from RAM almost instantly, while reaching all the way to disk is far slower. And it does this constantly. For every word the model writes, it reads through all those numbers.

From RAM that is quick. From disk, painfully slow. So how much RAM does it need? Roughly the size of the model.

An eight gigabyte model wants about eight gigabytes of free RAM, plus a little headroom. And it is the whole model at once. Every weight stays sitting in RAM the whole time it runs, not sipped in a bit at a time. Which leads to the catch.

The model has to fit. If the numbers need more RAM than your machine has free, they will not all fit. When it does not fit, your system pushes the overflow back onto the slow disk. This is called swapping.

Now it reads from disk again, and the model crawls. So, to run, a model is copied from disk into fast RAM, and it must fit. Keep free RAM near the file size and it stays fast. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is quantization? Let's try to understand. In a simple way. A model is millions of numbers.

And by default, each one is stored with lots of precision. A value like zero point three one four nine, carrying many digits. But here is the thing. All that precision is more than the model needs.

The exact trailing digits barely change what it does. So quantization is really just rounding. It replaces each precise number with a nearby, simpler one. Fewer digits, less fuss.

Here is the trick. Instead of allowing every possible value, you pick a small set of allowed values. A grid. Then you snap each number to the nearest one.

And because there are only a handful of allowed values, each number can be named with far fewer bits. That is the whole point of the word quantization. Take our number, zero point three one four nine. On a small grid, it might snap to just zero point three.

You store which slot it landed in, not the long decimal. The clever part is that the snapped value sits right next to the original. Close enough that the model barely notices the difference. Think of rounding every price to the nearest dollar.

Or a photo saved with fewer colors. Simpler numbers, and the picture still reads the same. So, quantization means making the model's numbers simpler. Snap each one to a small grid of allowed values, and store it with fewer bits.

Quick check now. One question is coming up. Let's see if it clicked.

How does quantization shrink a model's size? Let's try to understand. In a simple way. A model's file size comes from one simple recipe.

Take the number of weights, and multiply by the bytes each weight takes. That product is the size on disk. Here is the key idea. Quantization never removes a single weight.

A seven billion weight model still has seven billion afterward. The count stays fixed. So what actually changes? Only the second number.

The bytes each weight needs to be stored. Quantization makes every weight smaller on disk. By default, each weight is sixteen bits wide. That is two bytes apiece.

Seven billion of them lands you around fourteen gigabytes. Now drop each weight to eight bits. That is one byte. Same seven billion weights, but the file just halved, to about seven gigabytes.

Go further, down to four bits. Two weights now share a single byte. The very same model shrinks again, to roughly three and a half gigabytes. Notice the pattern.

Halve the bits, and you halve the file. The size scales straight down with the bits per weight. Nothing else moved. And that is why a model that would never fit suddenly does.

Same weights, same brain, just packed into fewer bits each. It slides onto your laptop. So, quantization shrinks a model by storing each weight in fewer bits. The weight count never changes.

Fewer bits, and a much smaller file. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the cost of a smaller model? Let's try to understand. In a simple way. Shrinking a model saved you a lot of memory.

But it is not completely free. Rounding every weight comes with a small price, and here it is. Quantization snaps each weight to the nearest value on its grid. That snap is never exact.

The stored number sits a hair off the true one. One weight off by a hair is nothing. But a model holds billions of weights, all rounded. Those tiny errors pile up as the layers stack.

So the shrunk model drifts a little. Ask the same question, and the answer can come out slightly less sharp. In effect, it gets a touch less accurate. How much less accurate?

You run the smaller model on a fixed test set and score it. That quality number tells you exactly what was lost. At eight bits, the drop is almost invisible. The model answers essentially the same as before.

For most work, that cost is basically nothing. At four bits, you see a small but real dip in quality. Still, for the huge memory it saves, that trade is usually worth taking. But push too far, down to two or three bits, and quality falls off a cliff.

The grid gets so coarse the errors finally overwhelm the model. So there is a sweet spot. You trade a little accuracy for a lot of saved memory. Shrink it smartly, and the cost stays small.

Quick check now. One question is coming up. Let's see if it clicked.

How do you decode a quantized model name? Let's try to understand. In a simple way. On a download page, a model shows up like this.

Llama three, eight B, Q four K M, dot GGUF. It looks cryptic, but every piece means something. Let us split the name into four parts. The family, the size, the format, and the quantization.

The first part is the family. Llama, or Mistral, or Qwen. That is who built the model, and which generation. Think brand and year.

Next comes the size. Eight B means eight billion weights. That is the rough scale of the model. Bigger means a heavier brain.

Then the ending, dot GGUF. That is the file format. One single file, packing the weights and settings together, built to run on your own machine. Now the interesting part.

Q four. The Q means quantized, and the four is the bits per weight. A smaller number means a smaller, lighter file. After that, letters like K and M.

The K marks the newer packing method. The last letter is a size tier. S small, M medium, L large. So read it left to right.

A Llama model, eight billion weights, GGUF format, squeezed to four bits, medium tier. The name just described the file. This is why the name matters. From those few symbols, you can guess the file size, and whether it fits, before you download.

So, a model name is really a label. Family, size, format, quantization. Read those, and you know what to download. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is a model inference engine? Let's try to understand. In a simple way. When you download a model, what you get is a file.

Billions of numbers, just sitting on your disk. On their own, those numbers do nothing at all. To turn a question into an answer, some program has to load those numbers and do the math with them. That program is the inference engine.

Think of the weights as sheet music. Silent on the page. The inference engine is the musician who reads it and turns it into actual sound. Its first job is loading.

The engine reads the weight file off your slow disk and places it into fast memory, ready to be used. Then you type a prompt. The engine turns your words into numbers and pushes them through every layer of the model, doing all the multiplications. Out comes a score for each possible next word.

The engine picks one, adds it to the text, and runs the whole thing again. One word at a time. You have probably seen their names. Ollama, and llama dot cpp, and a few others.

Different programs, all doing this same one job. And this is why the engine matters. The same weights on a better engine can run faster, using less memory. A good engine also knows how to read a compressed file, like GGUF.

So, the weights are the knowledge, and the inference engine is what brings them to life. It loads the file, runs the math, and writes the answer for you. Quick check now. One question is coming up.

Let's see if it clicked.

Why are GPUs faster for AI? Let's try to understand. In a simple way. Running an AI model is really just math.

The same tiny step, a multiply then an add, done billions of times over. A normal processor, a CPU, has a handful of large, powerful cores. Each is brilliant at complicated tasks taken one after another. But there are only a few.

A GPU is built the opposite way. Thousands of small, simple cores. Each one is weak, but together they work all at the very same time. Here is the key.

Those multiply and add steps do not depend on each other. Each can happen at the same instant. Work like that is called parallel. So a GPU hands one multiplication to each core, and they all fire together.

Thousands of answers appear in the time a CPU finishes a few. Picture a few math professors versus a stadium of students. For one hard proof, the professors win. For a million easy sums, the crowd wins.

All those cores are hungry for numbers. So a GPU also has a very wide memory pipe, feeding every core at once. A GPU is not magic. Give it one long chain, where each step needs the last, and it cannot spread the work.

AI just is not that shape. So GPUs win because AI is a mountain of identical, independent multiply and add steps. A CPU marches through them, a GPU does thousands at once. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the VRAM your model needs? Let's try to understand. In a simple way. VRAM is the fast memory on your graphics card.

It is separate from your regular RAM. And to run on the GPU, your model has to fit inside it. So the whole question is a comparison. On one side, how much VRAM the model needs.

On the other, how much your card has. Start with the weights. It is the same math as the file size. Take the number of parameters, and multiply by the bytes each one takes.

Say seven billion parameters. At two bytes each, that is about fourteen gigabytes. Shrink each weight to half a byte, and it drops under four. But the weights are not the whole bill.

While it runs, the model also holds your growing conversation. Longer chats need more room. So you add some headroom. A rough rule of thumb, take the weight size, and add about twenty percent on top for everything else.

Now you compare. If that total sits under your card's VRAM, it fits. The whole model runs on the GPU, at full speed. If it does not fit, the extra spills over onto slower memory, and the whole thing crawls.

Sometimes it simply refuses to load. The easiest lever is quantization. Fewer bits per weight shrinks the need, so a bigger model fits onto the very same card. So, VRAM is your GPU's memory budget.

Add up the weights, add a little headroom, and check the total fits on your card. Quick check now. One question is coming up. Let's see if it clicked.

How does your first chat with a local AI work? Let's try to understand. In a simple way. Here is the surprising part.

There is no server, and no account. The model lives on your own computer, and the whole conversation stays there. Before the first chat, you download the model once. It is a single file of weights, saved to your disk.

You never fetch it again. Then you open a chat window, or a terminal, and type a question. It looks just like talking to a cloud chatbot. The very first reply feels slow.

Before answering, the model must be lifted off your disk and into memory. People call this the cold start. Once it is loaded, the answer appears word by word, streaming out in real time. That flow is the model writing, right on your hardware.

You can keep going. Ask a follow up, and it remembers what you both just said. The conversation is held in memory for this session. One catch.

You pick a model small enough to fit your machine. A small local model is fast, but not as sharp as a giant cloud one. And it runs with the internet switched off. There is no live web behind it, so it only knows what it learned up to its training cutoff.

So, a first local chat is simple. Download once, type your prompt, wait for the load, and watch it answer, without ever leaving your computer. Quick check now. One question is coming up.

Let's see if it clicked.

How does a local model protect your data? Let's try to understand. In a simple way. With a cloud AI, your question does not stay with you.

It leaves your device, travels across the internet, and lands on a company's computer far away. Once your words sit on that machine, you no longer control them. They can be logged, stored for months, read by staff, or used to train the next model. A local model is different.

The weights already live on your own disk, and all the math runs on your own chip. So your prompt goes from your keyboard, into your own memory, and back to your screen. It never crosses the network at all. Here is the simple proof.

Turn off your internet completely. A cloud tool goes dark, but the local model still answers. Nothing had to leave for it to work. This is what protects sensitive things.

Medical notes, legal contracts, private code, a personal journal. They stay on the one machine you already trust. It also makes the rules easier. When data never touches an outside server, many privacy laws simply do not apply, because no third party ever held it.

One honest catch. Local means your input stays home, but the app around the model could still phone home. So pick trusted, open tools, and watch the network to be sure. So, local models protect your data by never sending it away.

The computation comes to your data, instead of your data going to the computation. Quick check now. One question is coming up. Let's see if it clicked.

How does talking to your documents offline work? Let's try to understand. In a simple way. You have a folder of your own files.

Notes, PDFs, a stack of contracts. You want to ask a plain question and get an answer straight from them. But a model can only read so much text at once. A thousand pages will never fit inside a single prompt.

So you cannot just paste it all in. The fix starts offline. Your files get chopped into small passages, about a paragraph each. This happens once, on your own machine.

Each passage is turned into a list of numbers that captures its meaning. That list is called an embedding. Similar meaning gives similar numbers. All those number lists are saved in a local index on your disk.

Think of it as a searchable library of your own writing. Now you type a question. It gets turned into numbers the very same way. One short list that stands for what you meant.

The index then finds the passages whose numbers sit closest to your question. The handful of paragraphs most likely to hold the answer. Those few passages are pasted in front of your question, as context. The prompt now says, here are the relevant notes, use them.

Your local model reads that packed prompt and writes an answer grounded in your files. This whole trick has a name. Retrieval augmented generation. And every step ran on your machine.

Chopping, embedding, searching, answering. Nothing was ever sent away, so no internet was needed at all. Quick check now. One question is coming up.

Let's see if it clicked.

What happens when your OS has AI? Let's try to understand. In a simple way. For years, if an app wanted AI, it had two choices.

Bundle its own model, or send your text off to the cloud. Every app on its own. Now the operating system itself carries a model. Baked into the system, next to the parts that handle your files and your screen.

So the model becomes a shared service. An app no longer brings its own. It just asks the system, like it asks for the camera. And it is loaded only once.

One model sits in memory, and every app shares that same copy. Not a separate one inside each program. An app calls it through a system interface. Rewrite this note.

Summarize this page. The answer comes back from the OS. All of this runs on a chip inside your device built for AI, called a neural engine. So it is quick, and it sips little battery.

Because the work stays on the device, your words never have to leave it. The system does the thinking right where you typed. And since the model lives in the OS, it shows up everywhere. Mail, notes, the keyboard, photos.

One brain, behind many apps. For a truly heavy request, the system can hand off to a larger model in the cloud. But the everyday small stuff stays local. So an AI in your OS is one shared model, loaded once, that every app can call.

On your device, and always within reach. Quick check now. One question is coming up. Let's see if it clicked.