Diffusion models, properly

What exactly is the forward diffusion process? Let's try to understand. In a simple way. A diffusion model learns to create pictures.

To teach it, we first go the other way. We start with one clean, real image. Then we add a small amount of random noise. A gentle sprinkle of static over the picture.

It still looks almost the same. Now we repeat that step. Add a bit more noise. Then a bit more.

Each step is tiny, but they stack up. After enough steps, the picture is gone. What remains is pure random noise. Static, with no trace of the original.

The static we add is Gaussian noise. Each pixel is nudged by a random number from a bell curve. Small nudges common, large ones rare. This is a Markov chain.

Every step depends only on the image right before it. Not on the clean start. Here is the key. This destroying process is fixed.

It has no knobs to train. The noise at each step is set in advance. So why wreck a good image on purpose? Because it gives us pairs.

A noisier version beside a cleaner one, at every step. Later, the model learns to reverse this. Undo the noise, one step at a time. But first, we needed a clean way to add it.

So, the forward process slowly turns an image into noise. Small Gaussian steps, a fixed recipe, no training at all. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is noise prediction? Let's try to understand. In a simple way. We want to run diffusion backward.

To walk from pure noise back to a real picture. For that, we need a network that can look at a messy image and clean it. Here is the clever part. Instead of asking the model to paint the clean image, we ask it to predict the noise.

The exact static that was added. Why the noise? Because we added it ourselves in the forward process. So we know exactly what it was.

That gives us a perfect answer key. So here is one training step. Take a clean image, add a known blob of noise, and hand the noisy result to the network. Let it guess.

Now compare. The model's guessed noise against the real noise we added. Their difference is the error, measured as mean squared error. Then nudge the network's knobs to shrink that error.

Repeat this over millions of images, at every level of noise. Slowly, it gets good. But why guess the noise, and not the clean picture? The two are equivalent.

Subtract the predicted noise from the messy image, and the picture appears. Yet noise is the easier, steadier target. After all that training, the network becomes a noise detector. Show it any noisy image, and it points at the static to remove.

So, we train the model by giving it noisy images and asking for the noise. Its guess is scored against the noise we know we added. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a U-Net? Let's try to understand. In a simple way. To reverse diffusion, the network reads a noisy image and predicts the noise to remove.

That output must give one value per pixel. This is not like labeling a photo cat or dog. A classifier gives one answer for the whole image. We need the same size out.

The U-Net earns its name from its shape. The image first travels down, getting smaller. Then back up to full size again. A U.

On the way down, each step shrinks the image but widens the view. A smaller map lets each part see more of the picture. At the very bottom sits the bottleneck. The image is tiny, but the network finally grasps the big picture, the overall shapes.

Now the way up. Step by step, the network expands the map back toward full resolution, painting details back in. But squeezing everything through that tiny bottleneck costs us. Fine detail, sharp edges and exact pixel positions, gets blurred away.

Here is the trick that saves it. Skip connections copy the sharp features from each downward level straight across to its match on the way up. The climb fuses the big picture with the rescued detail. And that is why it fits denoising.

Noise hides at every scale, coarse blotches and fine grain. The U-Net catches both, and returns a full-size noise map. So, a U-Net shrinks an image to grasp the whole, then rebuilds it, with skip connections rescuing the fine detail. Same size in, same size out.

Quick check now. One question is coming up. Let's see if it clicked.

How does the model know the noise level? Let's try to understand. In a simple way. To reverse diffusion, we call one single network again and again, at every noise level, from a wall of pure static down to a nearly clean image.

But here is the catch. Wiping a faint whisper of noise is a very different job from clearing a wall of static. Yet the same weights must do both. So the network needs one extra clue.

How far along the reverse process it is. Which step, out of the total. That number is the timestep. Every noisy image carries this timestep.

A large one means very noisy, early in the walk back. A small one means almost clean, nearly done. But you cannot just staple a bare number, like seven hundred, onto the image. To a network, a lone integer is a weak, clumsy signal.

So we expand that timestep into a rich vector, using sine and cosine waves at many frequencies. The same trick as positional encoding. Each step gets its own fingerprint. That time vector is then added into every block of the U-Net.

The encoder, the bottleneck, and the decoder. Every layer feels the clock. Now one set of weights can flex. At a high timestep, expect heavy noise and wipe boldly.

At a low one, tread gently and protect fine detail. So, the model knows the noise level because we tell it. The timestep becomes a wave vector, fed into every layer, so one network can denoise at any level. Quick check now.

One question is coming up. Let's see if it clicked.

How does a noise schedule work? Let's try to understand. In a simple way. The forward process turns an image into noise, one small step at a time.

But a question remains. How much noise should each step add? The answer is a schedule. A preset list of numbers, one per step, saying how strong the noise at that step should be.

Each of those numbers is called a beta. A small beta means a gentle sprinkle. A large one means a harsh burst of static. The schedule ramps up.

Tiny betas at the start, so the image is barely touched. Bigger betas near the end, until only pure noise is left. Now a handy trick. Multiply how much signal survives each step, from the start.

That running product gives the total noise at any step. This means we can skip the loop. Using that product, we jump straight to any step and add the right noise in one shot. So what shapes has this schedule taken?

The first was linear. The betas simply grow along a straight line, small to large. Later came the cosine schedule. A gentle curve that adds noise more slowly through the middle, not wiping the image out too early.

One last point. Like the forward process, the schedule is fixed in advance. A design choice, not something the network learns. So, a noise schedule is the preset recipe of how much noise each step adds.

The betas set the strength, and their running product lets us jump to any step. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the schedule's effect on image quality? Let's try to understand. In a simple way. Take one trained model and one dataset.

Change nothing but the noise schedule. The pictures can come out sharper or blurrier. The shape matters. Think of the steps as a budget.

The schedule decides where that budget goes. On the early gentle stage, or the late heavy one. The old linear schedule ramps up hard. By the middle, the image is already almost pure noise.

So the final steps work on static, and learn little. Those near noise steps are wasted. Fewer steps do real work, so the model has less chance to refine detail. The result is a softer, lower quality image.

A cleaner way to see it. At each step, look at the signal to noise ratio. How much image is left. The schedule really plans that ratio.

The cosine schedule keeps signal around far longer. It destroys the image slowly through the middle. So more steps carry useful information, and quality measurably improves. There is a catch at the end.

The last step should reach nearly pure noise. If a trace of image survives, sampling starts from something the model never expects, and quality drops. So it is a balance. Too gentle wastes early steps.

Too harsh wastes late ones. The best shape spreads useful signal across as many steps as possible. So, the shape decides where the real work happens. Spread the signal well, like the cosine curve, and the same model paints a cleaner picture.

Quick check now. One question is coming up. Let's see if it clicked.

How does one denoising step work? Let's try to understand. In a simple way. By now, the network is a trained noise detector.

To make a picture, we run the whole process in reverse. We start from a frame of pure random noise. But we do not leap straight to a clean image. We take one small step.

From the current noisy frame, to one that is just slightly cleaner. The step begins with a question. We hand the current frame to the U-Net, and it points at the noise it believes is hiding inside. Now the key move.

We do not subtract all of that guessed noise. We remove only a small, measured slice of it. Nudging the frame a little toward clean. Then a quick rescale.

We stretch the pixels back up by a fixed factor, so the overall brightness stays steady and does not fade. Here is the surprise. After that cleaning, we sprinkle a little fresh noise back in. On every step except the very last one.

Why add noise while denoising? Because it keeps the frame looking like something the model was trained on. Skip it, and the samples grow dull and collapse. Put it together, and that is one reverse step.

Predict, subtract a slice, rescale, then add a touch of noise. Our best guess for the previous frame. So one step back is small and careful. Point at the noise, shave off a sliver, and jitter it a little.

Then repeat, again and again. Quick check now. One question is coming up. Let's see if it clicked.

How does denoising in one jump work? Let's try to understand. In a simple way. The network is a trained noise detector.

Here is a tempting shortcut. Let it point at all the noise, and subtract every bit at once. So we try it. We start from pure random noise and leap to the clean image the network imagines.

But out comes a smudge, a washed out blur. Why so blurry? From pure noise, countless different clean images all fit equally well. A cat, a car, a face.

The frame is deeply ambiguous. So the network cannot pick one. Its single best guess becomes the average of every possible image. The average of many pictures is a gray, featureless blur.

Picture it another way. The path from noise to a sharp image is a curve, not a straight line. One big jump flies off on a tangent. The fix is many tiny steps.

After each one we hand the frame back and ask again. The network re-aims, and the path bends along with us. Step by step, the ambiguity shrinks. Early on, only the broad layout is fixed.

As the frame grows cleaner, fewer images fit, so the guess sharpens and detail appears. So a single jump trades quality for speed. One shot is instant but blurry. Many small steps are slower, yet they give a real, sharp picture.

So we cannot leap. From pure noise, one guess is only a blurry average. Small steps let the network re-aim, until the ambiguity collapses into one sharp image. Quick check now.

One question is coming up. Let's see if it clicked.

How does taking bigger sampling steps work? Let's try to understand. In a simple way. The plain sampler is slow.

It crawls back from noise in a thousand tiny steps, and every step calls the whole network. So here is the goal. Take far fewer, far bigger steps. Maybe twenty stops instead of a thousand, and still land a sharp picture.

The trick starts with a bold guess. From the current noisy frame, the network's noise estimate lets us jump to a full picture of the finished image, called x zero. Now the clever part. Instead of stepping to the very next level, we re-noise that clean guess down to any lower level we choose.

One move skips many rungs. This is DDIM. One more change. We drop the random jitter the slow sampler added each step.

The path turns deterministic. The same starting noise always lands the same image, so big skips stay stable. But the road from noise to image is a curve. A big straight step overshoots the bend.

Higher-order solvers peek ahead, so each large step still aims true. We also choose where to stop. A short list of noise levels, clustered where the image changes fastest, spends our few steps where they matter. The payoff is speed.

Twenty to fifty steps replace a thousand. Roughly twenty to fifty times faster, for nearly the same sharp result. So bigger steps are a recipe. Predict the clean image, jump to a chosen lower level, go deterministic, and let a curve-aware solver keep it sharp.

Quick check now. One question is coming up. Let's see if it clicked.

How does steering diffusion with a classifier work? Let's try to understand. In a simple way. The plain sampler starts from noise and denoises its way to a picture.

But you get whatever it feels like drawing. You cannot ask for a cat. Say we want a cat, not just any image. We need a way to push every step toward cat, and away from everything else.

So we bring in a second model, a classifier. It looks at an image and scores how much it looks like a cat, a dog, a car, and so on. But here is the catch. During sampling the image is still noisy.

So this classifier has to be trained on noisy images, at every noise level, not on clean photos. At each step we ask the classifier one question. Which way should the pixels move to look more like a cat? That direction is a gradient.

Then we add that gradient into the denoising step. The denoiser still removes noise, but now each step also leans a little toward cat. A single dial sets how hard we push. Turn it up and the cat gets stronger and clearer.

Turn it down and the model stays freer. The price is a whole extra model. That noise aware classifier has to be trained separately, just to hand back its gradient at every step. So classifier guidance steers with a second model.

Score the target class, follow its gradient, and nudge every step toward the image you asked for. Quick check now. One question is coming up. Let's see if it clicked.

How does guiding diffusion without a classifier work? Let's try to understand. In a simple way. Classifier guidance needed a whole extra model.

A separate classifier, trained on noisy images, just to point each step toward the prompt. That is baggage. So here is the trick. What if the model guides itself?

No outside judge. The same network points toward the prompt on its own. The secret is in training. Sometimes we hide the prompt and feed an empty label.

So one network learns two jobs. Denoise with the prompt, or with nothing. At sampling we run the network twice. Once with the prompt, once with the empty label.

We hold two noise guesses. One aimed at a cat, one at anything. Subtract one from the other. The gap between the two guesses is a direction.

It points from a generic image toward what you asked for. That is the guidance direction. Then we push. Start from the plain guess and step along that gap, further than the prompt alone would go.

That overshoot becomes our noise estimate. A single dial sets how far we push. It is called the guidance scale. But there is no second model.

The same U-Net does both passes. So what does it cost? Not a whole extra network. Just two passes through the same one every step.

Twice the compute, no classifier to train. So classifier free guidance drops the outside judge. Train one model with and without the prompt, subtract the two guesses, and push along that gap. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the guidance scale? Let's try to understand. In a simple way. Remember the guidance gap.

It points from a generic image toward your prompt. But how hard should we push along it? One number decides. That number is the guidance scale.

It multiplies the gap. At a scale of one, you take the prompt's own guess. Turn it higher, and you push further. Turn the dial low, and the push is gentle.

The image barely listens to your prompt. It stays generic and varied, but it could draw almost anything. Turn the dial high, and the push is forceful. Now the image locks hard onto your prompt.

Every detail you asked for is pulled to the front. So here is the real trade. More scale obeys the prompt, but colors grow over saturated and contrast blows out. Less scale looks natural, but drifts off topic.

Push too far and it breaks. The overshoot lands on images the model never really learned. You get garish colors, harsh edges, and odd repeated patterns. So people pick a middle value.

For many image models, something around seven or eight. Strong enough to follow the prompt, gentle enough not to burn it. One warning. This is not a quality knob.

Higher does not mean smarter or better. It only sets how strongly the prompt overrides the generic pull. So the guidance scale is one number that multiplies the gap. Low is free but vague, high is obedient but harsh, and the middle is the sweet spot.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is latent diffusion? Let's try to understand. In a simple way. A picture is a huge grid of pixels.

Running every denoising step on all of them is slow and heavy. Hundreds of steps, millions of numbers each time. So here is the trick. Do not diffuse in pixel space at all.

First squeeze the image down into a smaller, hidden space. An encoder does the squeezing. It is one half of an autoencoder. A big grid of pixels goes in, a small grid of numbers comes out.

That small grid is the latent. Why can we shrink it? Because most pixels are redundant. Neighbors look alike.

The latent keeps the meaning and throws the wasted detail away. Now run the whole process right there. The forward noising and the denoising both happen on the tiny latent. The U-Net never touches raw pixels.

The payoff is huge. Far fewer numbers per step means each step is lighter. Compute and memory drop sharply. This is why such models can run on a single graphics card.

When denoising finishes, you hold a clean latent, not a picture yet. So a decoder, the other half, expands it back into full pixels. One catch. The encoder and decoder are trained once, then frozen.

They handle fine texture, while diffusion handles the meaning. Tiny details can soften, but the trade is worth it. So latent diffusion compresses first, diffuses in the small hidden space, then decodes back to pixels. The same idea, at a fraction of the cost.

Quick check now. One question is coming up. Let's see if it clicked.

How do text prompts steer the U-Net? Let's try to understand. In a simple way. The U-Net we built is a great denoiser.

But on its own, it steers toward just some image. Nothing tells it what you actually want. You want control. So you type a prompt.

A red fox in snow. Now those words have to get inside the network somehow. First a separate, pretrained text encoder reads the prompt. It turns the words into a list of vectors, one per token.

Meaning, written as numbers. Those vectors are not fed in once at the start. They are handed to every block of the U-Net, so the text and the image meet again and again. That meeting has a name.

Cross-attention. The image features form the queries. The text vectors form the keys and the values. So each patch of the noisy image looks across all the words, finds the ones that matter to it, and pulls their meaning in.

Now the noise the U-Net predicts depends on the prompt. Change the words, and the noise changes, and so does the picture it builds toward. Why every layer? Early blocks lay down big shapes, later blocks add fine texture.

The prompt gets a say at every scale. So the prompt becomes vectors, and cross-attention lets every patch read them. The words quietly steer each denoising step. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the gradient view of denoising? Let's try to understand. In a simple way. We have been denoising step by step, subtracting a little predicted noise each time.

But there is a deeper way to see what those steps are really doing. Picture a landscape of all possible images. Real, natural pictures pile up into tall hills. Random noise lies far away, down in the empty flatlands.

Every denoising step nudges the image a little uphill, toward where real images are dense. Follow that slope and you climb from noise back to a believable picture. At every point in this landscape there is an arrow pointing uphill. The network's job is to read your noisy image and hand you that arrow.

That uphill arrow has a name. The score. It is the gradient of the log probability, the steepest direction toward more likely images. Here is the twist.

The noise the network predicts is that same arrow, just flipped around. Subtracting the predicted noise is exactly one step along the score. So sampling is a walk. Read the slope, take a small step uphill, add a touch of fresh randomness, and repeat.

That is why we say it follows a gradient. Why care? This single idea links noise prediction to score based models. It is also why you can add other gradients, like a prompt, to steer the same walk.

So denoising is really gradient ascent. The predicted noise points uphill on a landscape of likely images, and sampling follows that slope home. Quick check now. One question is coming up.

Let's see if it clicked.