Diffusion Models, dissected

What exactly is a generative model? Let's try to understand. In a simple way. Start with a pile of training photos.

Some images are likely, most are not, and a hidden rule decides which. That rule is the data distribution. Every training image is one draw from it. Picture a lottery over all images, where bedrooms win often and static almost never does.

A generative model is a machine with a lever. Each pull produces a fresh image. Training tunes its knobs, called parameters, until the draws look like the data. Concretely, a thirty two by thirty two color image is three thousand seventy two numbers, each from zero to two hundred fifty five.

The distribution covers all of them. How do we grade the machine? Yardstick one is likelihood: the probability the model gives real images it never saw. High probability means they do not surprise it.

We report it as bits per dimension, the bits needed to describe each number. Three bits per dimension is about nine thousand bits per image. Lower is better. Yardstick two looks at the draws themselves.

Generate fifty thousand images and score how close they look to real ones. This paper uses Inception score and FID. FID compares the statistics of the generated batch with a real batch. Identical batches score zero, so lower is better.

Inception score runs the other way: higher is better. Keep both, because they disagree. A model can draw gorgeous images yet give real ones low probability. This paper lands on exactly that split.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is one step of Gaussian noise? Let's try to understand. In a simple way. Last time, training images were samples from a distribution.

The simplest distribution to draw from is the Gaussian, a bell shaped pile of numbers with two knobs. The mean is where the bell sits. The variance is how wide it spreads. With mean zero and variance one, most draws land between minus two and plus two.

Now take an image. Every pixel is a number. Give each pixel its own Gaussian draw, independent of its neighbors, and add it in. It turns grainy.

The rule that matters: add two independent Gaussian quantities and their variances add. Variance one plus noise of variance one gives variance two. The image gets louder. So the paper's step shrinks first.

Multiply every pixel by the square root of one minus beta, with beta a small positive number. Then add noise with variance beta. Now the bookkeeping. Scaling by a factor scales variance by the factor squared.

So variance one becomes one minus beta. Add the noise's beta: back at exactly one. Take beta as one hundredth. The shrink factor is the square root of ninety nine hundredths, about zero point nine nine five.

Each pixel shrinks half a percent, then gains noise of variance one hundredth. Skip the shrink and every dose of noise would inflate the variance. With it, the noised image, given its starting image, is Gaussian: mean the shrunken image, variance beta. One step.

Next lesson, many steps. Quick check now. One question is coming up. Let's see if it clicked.

How does the forward chain to pure noise work? Let's try to understand. In a simple way. Last time we took one step: shrink the image slightly, add a little Gaussian noise.

Now repeat it one thousand times. That is a Markov chain: each noisy image depends only on the one before it. It is fixed. Nothing is learned here.

Each step has its own beta, its noise amount. The schedule is linear: one ten-thousandth at step one, rising in a straight line to two hundredths at step one thousand. Pixels are scaled between minus one and one, so even two hundredths is tiny. No single step changes much.

The count does the damage. Now the shortcut. Call one minus beta alpha. Multiply the alphas from step one to step t; that product is alpha bar.

It tracks how much clean image survives. Gaussian noise on Gaussian noise stays Gaussian, so you can skip the chain. Scale the clean image by the square root of alpha bar, add noise scaled by root one minus alpha bar. One draw, any step.

Run that to step one thousand. Alpha bar collapses to almost nothing; its square root is under one percent. That is all of the picture that survives. The paper checks this: the gap between step one thousand and pure Gaussian noise is about one hundred thousandth of a bit per dimension.

Essentially zero. We have built a machine that only destroys. Any image in, static out. Nothing learned.

Next, we run it backwards. Quick check now. One question is coming up. Let's see if it clicked.

How does the reverse process work? Let's try to understand. In a simple way. Last lesson, we walked an image forward, a thousand small steps, into pure static.

Generation runs that walk in reverse: from fresh static, step back toward a picture. Forward steps were fixed recipes. Backward steps must be learned: given the noisy image at step t, produce a slightly cleaner one at step t minus one. Undoing one big noise step could give a wild, lumpy distribution.

But when every forward step is tiny, the reverse step keeps the same shape: Gaussian. A Gaussian needs only a center and a spread. The paper pins the spread to a fixed constant, so the network's whole job is the center, one number per pixel. What should that center be?

In training we built the noisy image from a clean one. Knowing both, the one-step-cleaner image has a known Gaussian, one exact formula. Its center blends noisy image and clean image, weights set by the schedule. Where alpha bar is about one half, that is about ninety nine percent noisy and under one percent clean.

That just means one step barely moves. Its spread is fixed by the schedule too, a bit under beta. A thousand tiny moves make the trip. In training the clean image is in hand, so the ideal step is fully known.

At generation time there is none; the network's center stands in. The network proposes a Gaussian. The truth is a Gaussian. Next lesson scores how far apart two Gaussians are.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the variational bound? Let's try to understand. In a simple way. Two Gaussians per step: the network's, and the true one-step-cleaner one.

KL divergence scores how far apart two distributions sit. Zero when they match, positive otherwise. For two equal-variance Gaussians, KL has a closed form: mean gap squared, over twice the variance. Means one tenth apart, variance one hundredth: one hundredth over two hundredths.

One half. Now the bound. Steps two through one thousand: take that KL between the network's step and the true one, and add them up. Nine hundred ninety nine scaled mean gaps, squared.

Step one is different: pixels are whole numbers. So the decoder term is minus the log probability the final Gaussian puts in a small bin around each true pixel value. One term remains: step one thousand against pure noise. Nothing learned touches it.

The schedule is fixed, so it is a constant, about one hundred thousandth of a bit per dimension. Ignored. Why bits? Minus log probability, base two, is the bits a coder needs to write the data under your model.

Each term is a chunk of one message: the bound is a codelength. Why a bound? The exact cost sums over every path through a thousand noisy images. Impossible.

But this total, taken along the fixed forward chain, never dips below the true cost: a variational bound. Push it down, the true cost follows. So per step, the network's job is sliding its mean onto the true mean. What should it output?

Next lesson. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is noise prediction? Let's try to understand. In a simple way. Last lesson's loss scored the gap between the network's center and the true center.

Obvious design: output that center. The paper picks a better target. Recall the shortcut. Noisy image equals root alpha bar times clean image, plus root one minus alpha bar times noise.

Know two of the three, the third is forced. One pixel. Clean value zero point six, noise minus one, alpha bar one half, so both roots are about zero point seven one. Noisy value: about minus zero point two eight.

Given that and the noise, zero point six returns. The true center blends clean and noisy image. But clean is just noisy minus scaled noise, over root alpha bar. Substitute, and the clean image vanishes.

What remains: noisy image, minus beta over root one minus alpha bar times noise, divided by root alpha. Noisy image in hand, schedule fixed. The only unknown is the noise. So the network changes jobs.

Given noisy image and step number, it outputs a picture of the noise it believes was mixed in: epsilon theta. The center is no longer learned. Drop the predicted noise into the formula and the center falls out. Fixed arithmetic, nothing trained.

Why not predict the clean image and derive the noise? Same information. The paper tried it early and found worse samples. Noise won.

Rewrite the loss with this target: a weighted squared gap between true and predicted noise. That weight is the paper's next move. Quick check now. One question is coming up.

Let's see if it clicked.

How does the four-line training loop work? Let's try to understand. In a simple way. Last lesson the network learned to output the noise.

So how is it trained? The whole training algorithm fits in four lines. Line one. Pull a clean image from the training set.

Line two. Pick a step at random, one to one thousand, all equally likely. Line three. Draw fresh Gaussian noise, one value per pixel, mean zero, variance one.

This is what the network must guess. Mix them with lesson three's alpha-bar shortcut: root alpha-bar times the clean image, plus root one minus alpha-bar times the noise. One draw, no chain. Say alpha-bar is one quarter.

Root of a quarter is one half, root of three quarters is about point eight seven. Half the image, most of the noise. Line four. Feed the noisy image and the step number to the network.

It guesses the noise. Take the squared difference between true and guessed noise, and step the weights downhill. Repeat until the loss stops falling. Something is missing.

Last lesson's rewritten bound weights every step's squared error. This loss, called L simple, drops that weight. Those weights were largest at small steps, where barely any noise was added and the job is easy. Dropping them shrinks the easy steps' share, so the network works hardest on the noisy steps.

It is no longer the exact bound, just a reweighted one, picked by experiment. Training done. Next, we sample. Quick check now.

One question is coming up. Let's see if it clicked.

How does sampling from static work? Let's try to understand. In a simple way. Training is done.

Now generate. Start with pure Gaussian static, mean zero, variance one per pixel, at step one thousand. Each step, the network sees the image and the step number and predicts the noise inside. Subtract that prediction, scaled by beta over root one minus alpha bar.

Then divide by root alpha. The forward chain shrank the signal every step; this undoes it. With beta at two hundredths, the factor is about one point zero one. The surprising part.

Add fresh Gaussian noise with variance sigma squared. Adding noise while removing noise. Each reverse step is a Gaussian, and a Gaussian has spread. Repeat a thousand times, one network call each.

On the last step, skip the fresh noise. What comes out is the sample. Sigma is not learned. Two fixed choices: beta, or beta tilde, the variance of the true one-step-cleaner distribution.

Beta is right if the data is pure static; beta tilde, if it is one single image. Real images sit between those extremes, so the two choices bracket the truth. The paper tried both; results were similar. Langevin dynamics is an old sampler: nudge a point toward higher probability along the gradient, the steepest uphill direction, then add a small random kick.

Repeat. Look at our loop again. Subtract predicted noise, add a kick. The paper says this resembles Langevin dynamics, the predicted noise acting as a learned gradient: subtract it and you climb toward likelier images.

Quick check now. One question is coming up. Let's see if it clicked.

How does the D D P M network work? Let's try to understand. In a simple way. Until now the noise predictor was a black box: noisy image and step in, predicted noise out.

What did the paper, Denoising Diffusion Probabilistic Models, D D P M, build? A U-Net. It shrinks the image through smaller grids, then grows it back to full size, with shortcuts passing detail across at each size. For CIFAR ten, the thirty two pixel images, four grid sizes: thirty two, sixteen, eight, four.

Each holds two residual blocks, layers that add their output onto their input, learning only a correction. At the sixteen by sixteen grid comes self-attention: every position weighs every other, so a distant patch informs a local guess. One network serves all thousand steps, so it must know which. The step number becomes a sinusoidal embedding, sine and cosine waves at varied frequencies, added inside every residual block.

The CIFAR ten model has thirty five point seven million parameters, its learnable numbers. Dropout randomly zeroes some internal values during training so the network cannot memorize. They swept and kept one tenth; with none, samples got worse, with signs of memorizing. They also keep an exponential moving average of the weights, decay point nine nine nine nine, smoothing recent jitter.

Eight hundred thousand steps at batch one hundred twenty eight. On one TPU v three dash eight accelerator, twenty one steps per second: about ten and a half hours. A plain recipe. Whether it pays off, the numbers decide next lesson.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the D D P M scorecard? Let's try to understand. In a simple way. Last lesson was the recipe.

Now the question that matters. Did it work? Two sample scores. FID, from lesson one, lower is better.

And Inception score, which rewards clear, varied objects. Higher is better. On CIFAR ten, tiny thirty two pixel images, the model scores Inception nine point four six and FID three point one seven. Unconditional means no labels, just pixels.

Of every unconditional model listed, this FID is the lowest, even edging StyleGAN2, the strongest GAN listed, a rival family where a generator fools a detector. One caveat. Give StyleGAN2 the class labels, cat or truck, and its FID drops to two point six seven. With labels, the GAN stays ahead.

Now the big LSUN photos, two hundred fifty six pixels wide. Bedrooms: FID six point three six, against eight point three four for ProgressiveGAN, an older GAN. Cats too. But churches go the other way, and StyleGAN and StyleGAN2 win every column.

So the abstract claims parity with ProgressiveGAN, not victory over GANs. Then the ablation: change one design choice at a time, rescore. Mean or noise. Full bound or L simple.

Fixed variance or learned. Noise prediction on the full bound: FID thirteen point five one, roughly tied with predicting the mean. Switch to L simple: three point one seven. The pairing wins.

Letting the network output its own variance? Unstable training, poor samples, scores off the chart. Every reported model keeps lesson eight's fixed sigma. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the rate versus distortion split? Let's try to understand. In a simple way. On held out CIFAR ten this model needs three point seven five bits per dimension.

Sparse Transformer, which writes pixels one at a time, needs two point eight. The authors admit it is not competitive. Split the code. The bound is per-step terms plus a final decoder term.

Step terms are the rate, bits spent on the noisy path. The decoder term is the distortion, bits to pin down exact pixels. Rate, one point seven eight. Distortion, one point nine seven.

That distortion is a typical pixel error of zero point nine five on the zero to two fifty five scale. Under one gray level. Invisible. So more than half the code buys detail no eye can see.

The paper says so. Stream it. After five hundred reverse steps, seven thousandths of a bit spent, error thirty eight. After nine hundred, twelve hundredths, error twelve.

The last stretch spends almost everything. At any step, unscale the predicted noise into a guess of the clean image. During sampling, large shapes appear first, fine detail last. Push two faces forward to step five hundred, blend the noisy versions, run the chain back.

A face in between comes out. Blend later and it gets coarser; at one thousand, a new face. Blank one pixel per forward step instead of adding noise, and the reverse model is exactly the one-pixel-at-a-time kind. So Gaussian diffusion is that family, with an ordering no pixel shuffle can express.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the fine print on D D P M? Let's try to understand. In a simple way. The authors name what does not work yet.

Four caveats, then the family tree. First, speed. Every sample runs all thousand steps. A batch of two hundred fifty six images took seventeen seconds on their T P U.

No shortcut offered. Second, that F I D of three point one seven was measured against the training set, standard practice. Against held-out test images it rises to five point two four. Third, likelihood.

Lesson eleven's three point seven five bits trail Sparse Transformer's two point eight. Not overfitting: train and test differ by at most three hundredths of a bit. Fourth, the winning loss. Dropping the per-step weight was a choice that worked, not a result derived from the bound.

It raised the codelength slightly. Now the lineage. The forward-and-reverse chain is Sohl-Dickstein and colleagues, twenty fifteen. Estimating the data density's gradient at many noise levels, then Langevin sampling, is Song and Ermon, twenty nineteen.

Follow-ups took the caveats in turn. D D I M kept the training but sampled far faster. Improved D D P M learned the variances unstable here and reached competitive likelihoods. Score S D E folded the thousand steps into one continuous process, F I D two point two zero.

Then Diffusion Models Beat GANs, twenty twenty one, claimed the headline this paper never did. This course ends here. No prompts, no conditioning. Just a network that predicts noise, and honest numbers around it.

Quick check now. One question is coming up. Let's see if it clicked.