Creating with generative media

How does making an image from noise work? Let's try to understand. In a simple way. Every image generator starts the same way.

Not a blank canvas. A screen full of random static. Every pixel is just a random number. Here is the strange part.

The model never paints anything onto that screen. It only ever takes something away. All it does is remove noise. To learn that, it practised the opposite direction.

Take a real photo, sprinkle noise on it, and repeat until the photo is buried in static. That gives millions of pairs. A messy picture, and the exact noise that went in. So the network trains for one job.

Look at a mess, and point out the noise. Generation runs that backwards. Show the network pure static. It predicts the noise.

We subtract a small slice, and the screen is a little less random. Then we ask again, on the cleaner screen. And again. Roughly thirty passes, each one nudging the pixels closer to something that looks like a photo.

This is why it looks like emerging. Early passes settle the rough shapes and the layout. Late passes fill in edges, texture, and fine detail. One warning.

The picture was never hiding inside the noise. The static is only a starting point. What fills it in is the network's learned sense of how photos look. So an image emerges because a noise remover runs over and over.

Start from static, shave a little off, repeat, and a picture is what remains. Quick check now. One question is coming up. Let's see if it clicked.

How does text guidance work? Let's try to understand. In a simple way. Last time, the model turned pure noise into a picture.

But into which picture? Nothing in that loop said cat, or castle, or car. It just cleaned. So we hand it your words.

A separate network, the text encoder, reads your prompt and turns every piece of it into a vector. Meaning, written as numbers. Those vectors never get painted onto the canvas. They ride along beside it, like directions handed to the person who is doing the drawing.

Remember the one job. Look at the messy canvas, and predict the noise in it. Now the guesser holds your prompt vectors too, and its guess changes. That meeting is called cross attention.

Every patch of the canvas looks across all your words, finds the ones that concern it, and pulls their meaning in. And this happens at every step. Not once at the start. All fifty passes get the same words, so the picture is nudged again and again.

Here is the trick that gives text its force. At each step the model guesses the noise twice. Once with your words. Once with nothing at all.

The gap between those two guesses is exactly what your words are pulling for. The sampler follows that gap, and turns it up. That is guidance. So text never draws anything.

It only bends each guess about the noise, a little, fifty times over, until the picture that appears is the one you asked for. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the hidden space where concepts live? Let's try to understand. In a simple way. Start with the size problem.

Five hundred pixels across, three numbers each, and one photo is nearly eight hundred thousand numbers. Denoising that thirty times is brutal. So the model never works on pixels. A network called an encoder squeezes the photo down first.

Five hundred across becomes sixty four across, four numbers per cell. That is sixteen thousand numbers, not eight hundred thousand. Roughly fifty times smaller. This little grid is called the latent.

A matching decoder blows it back up. Here is the part that matters. The noise, and all the denoising, never touch your pixels. It happens inside the little grid.

The decoder paints at the very end. But the space is not just smaller. It is organized. Two golden retrievers land close together in there.

A bus lands far away. So you can walk a straight line between two points and decode every step. Not a crossfade. A smooth morph, every step still looking real.

Directions carry meaning too. Nudge a point one way, the face gains glasses. Nudge another way, it ages. Nobody labelled those axes.

That is the hidden part. One warning. The latent is not a tiny thumbnail. Raw, it is a smear of numbers.

It becomes a picture only when the decoder reads it. So, the model thinks in a small squeezed space where meaning lives. Encode, denoise in there, decode. That is what made image generation cheap.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the red cube problem? Let's try to understand. In a simple way. Ask for a red cube next to a blue sphere.

Two shapes, two colors. But what often comes back has the colors swapped. A blue cube, and a red sphere. Here is why.

Your prompt is not read as a sentence with rules. It becomes a bundle of numbers, and nothing in it staples red onto cube. While the picture is cleaned up, every patch of canvas looks at every word. The sphere's patches read red just as easily as the cube's patches do.

So the color leaks. Nothing says this adjective belongs to that noun. Researchers call this attribute binding, and it is where these models are weakest. The same crack breaks counting.

Ask for five apples, and you get four, or seven. There is no counter inside. It paints appleness until it looks right. Left of, and on top of, are just more words in the same pool.

They carry no coordinates. The model treats them as a hint, not an instruction. Why was this never learned? The training captions.

They say, a cat on a couch. Almost none say, three red cubes left of a blue sphere. And a red cube is rare. The model has seen many red balls and wooden cubes, so it drifts toward the familiar.

So ask for less. One object, one attribute. A red cube is hard because nothing ties a word to a thing. The prompt is a pool, not a sentence.

Quick check now. One question is coming up. Let's see if it clicked.

How does a negative prompt work? Let's try to understand. In a simple way. You asked for a portrait.

What came back has a watermark across the corner, and the whole thing is blurry. You never asked for either. So you do the obvious thing. You add, no watermark, to your prompt.

And it comes back with two of them. Here is why. Your prompt becomes a pool of word vectors. A pool has no way to say not.

You just dropped the word watermark into it. Now remember how guidance works. At every step the model guesses twice. Once holding your words.

Once holding nothing at all. That second slot has been sitting empty this whole time. A negative prompt fills it. Now the second guess is holding, blurry, watermark.

And the sampler steps along the gap between them. Your guess minus that one. So every step moves away from blurry, and toward your portrait. So nothing is ever removed.

There is no filter at the end scrubbing watermarks off the picture. It is a direction, pushed at every step. And it is free. That second guess was already running, every step, on nothing.

You just gave it something to say. But turn guidance up and the push gets harder. Shove too far from blurry, and you can lose soft backgrounds and smooth skin along with it. So, a negative prompt is not a delete button.

It is the second guess. You fill it, and the model walks the other way. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is prompt weighting? Let's try to understand. In a simple way. Ask for a misty forest with a red lantern.

Back comes forest, everywhere. The lantern is a speck. You want one word louder, not new words. So most tools give you a knob.

Write the word with a number beside it. Lantern, one point four. That number is a weight. What does that number do?

Every word in your prompt is already a vector of numbers. The weight multiplies that vector, stretching it longer. And a longer vector pulls harder. As each patch of canvas looks across your words, more of them decide they are lantern.

But weights are relative. Raise one word and every other word gets quieter beside it. Turn every word up to one point five, and nothing moves. Push it toward two and it burns.

The vector lands where the model never trained. Colors blow out, and the shape melts. Below one, the word fades. At zero it is simply not asked for.

That is not a negative prompt, which pushes the picture away. Do not confuse this with guidance. Guidance is one volume knob for the whole prompt. Weighting is a mixing desk, one slider per word.

It still cannot tie a word to a thing. Turn red up and you get more red everywhere. Not red on the cube. So a weight is a multiplier on one word's vector.

Nudge it, do not shove it. And the numbers only mean something next to each other. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a seed? Let's try to understand. In a simple way. Type the same prompt twice.

A knight in a forest. You get two different people. Same words, two strangers. So something changed, and it was not the words.

Every picture starts as a field of pure static, and the model cleans it up. That starting static is different on every run. Different static, different face. But a computer cannot really be random.

It runs a formula that spits out numbers which only look random. The formula needs one number to start from. That is the seed. Feed in the same seed and the formula lays down the exact same static.

Same seed, same prompt, same settings, and the same picture comes back. Pixel for pixel. So the seed is a bookmark. Lock it, change red coat to blue coat, and most of the picture holds.

The same face comes back, in blue. But the seed only repeats when everything else matches. Change the sampler, the step count, or the image size, and that same seed hands you a stranger. Here is what people get wrong.

The seed does not store your character. There is no face inside the number. It is a starting point, not a memory. So it is fragile.

Push the prompt far enough, from a forest to a spaceship, and the shared start stops carrying the face. In practice. Roll seeds until a face lands. Write the number down.

Then hold it still, and change as little as you can. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is image to image? Let's try to understand. In a simple way. Normally a run starts from pure static.

Nothing of yours in it. But you have a picture already. A first try, nearly right. You hand it in as the start.

Here is the surprise. The model never copies it. It wrecks it first. Your picture goes into the latent, and noise is poured on top.

Not all the way to static. Partway. Now it looks half finished. That picture drops into the middle of the denoise loop.

The early steps never run. The model wakes at step fifteen of thirty, and assumes it made the rest. One dial decides all of it. Strength.

It sets how much noise goes on, which is how far back you start. Two tenths is a whisper of noise. Only the last steps run, so your picture barely moves. Nine tenths buries it, and something new comes out.

What carries over? Only what the noise did not reach. The layout, the big shapes, the blocks of color. Fine detail dies first.

Your prompt still steers every step that is left. The picture says where things sit. The words say what they become. Dog into cat fails at low strength.

Too little noise to move the shape. You get a dog in cat colors. Turn it up and the cat arrives, but your dog is gone. So, image to image is not a blend.

Noise up to a point, then denoise from there. Strength is where you jump in. Quick check now. One question is coming up.

Let's see if it clicked.

How does copying a pose work? Let's try to understand. In a simple way. You have a photo with the exact stance you want.

Arms up, one knee bent. Now write that in words. You cannot. A pose is geometry.

So the picture goes to a preprocessor first. It throws the photo away and keeps a stick figure. Shoulders, elbows, hips, knees. Just the joints.

Here is the twist. The run still starts from pure static. Your photo is never handed in. Only that skeleton is.

The skeleton goes to a second network, sitting beside the main one. A copy of its encoder, trained to read maps like this. That is ControlNet. It does not speak once.

At every denoise step it adds a hint into the main model's layers. Put a shoulder here. An elbow there. So the two conditions split the job.

The prompt says what: a knight, gold armor. The map says where: this arm, this angle. One dial sets how loud the hint is. Turn it down and the model drifts off the pose.

Turn it up and limbs bend to fit, even when it looks wrong. Swap the extractor and you copy something else. A depth map copies distance. Canny edges copy outlines.

A scribble copies your rough shape. This is what people get wrong. Nothing of the photo's person survives. Not the face, the clothes, the colors, the light.

Structure only. So, trace the photo into a map. Start from static anyway. Then hand that map in at every step.

Quick check now. One question is coming up. Let's see if it clicked.

How does injecting your own face work? Let's try to understand. In a simple way. The model has seen millions of faces.

Never yours. And words cannot get you there. Brown eyes, short beard, round jaw. That is a thousand strangers.

A starting image or a pose map steers one run. Then it is gone. The model still does not know you. So you change the model itself.

Show it ten or twenty photos of you. Different light, different angles, same face. Every photo carries the same caption, holding one rare, made up word. The model has no use for that word yet, so your face lands on it cleanly.

Retraining every weight is too slow. So freeze them all. Bolt a small piece beside the attention layers, and train only that. That is a LoRA.

The model is gigabytes. That file is megabytes. Because the change is low rank. Not a full grid of edits.

Two skinny tables, multiplied together. Swap the photos and you swap what it learns. Feed one artist's pictures, and the word picks up the look. Strokes, palette, edges.

Not a person. After training, you hand in nothing. You type the word. Your face on a spaceship, from any angle.

It lives in the weights now. Train too long, or on too few photos, and it memorizes them. Same shirt, same background. Every face in the picture becomes yours.

So, a few photos, a rare word, a small patch of weights. Your face is not handed to the model. It is taught. Quick check now.

One question is coming up. Let's see if it clicked.

How does video generation work? Let's try to understand. In a simple way. The obvious way is one picture at a time.

Draw a frame, then the next, then the next. It fails. Every run rolls its own static, so nothing carries. The cat changes coat every frame.

So the model does not make frames. It makes the whole clip at once. Sixteen frames, born together, out of one block of static. The latent grows a third axis.

Not just width and height. Time. Picture a cube of numbers, one slice for each frame. Then the same denoise loop runs, on the cube.

Every step cleans all sixteen frames together. There is no first frame. They all arrive at once. But how does frame five know about frame one?

Attention reaches across time. Each patch looks at the same patch in the other frames. Those are the temporal layers. That is bolted onto an image model.

Keep the layers that already know what a cat looks like. Slot new time layers between them. Then train those on video. Motion is learned, never simulated.

There is no physics inside. It saw real clips of falling, walking, splashing. Dropped things fall because they fell in the training data. Which is why clips are short.

The cube is frames times everything. Sixteen pictures in memory at once, all looking at each other. A few seconds is expensive. So, a video is not a row of pictures.

It is one cube of static, denoised as a whole, with attention across time. Quick check now. One question is coming up. Let's see if it clicked.

How does smooth video motion work? Let's try to understand. In a simple way. Smooth and consistent sound like one thing.

They are two. Smooth is the motion between frames. Consistent is the thing staying the same thing. Watch a clip closely.

The shape holds, but the texture crawls. A shirt pattern rewrites itself. The frames agree on the big picture and argue about the details. So there is a dial.

Motion strength. Turn it down and the clip is rock steady, nearly a photograph. Turn it up and things move, and everything boils. Frames are expensive, so the model makes few.

Then a second model fills the gaps. It takes two real frames and asks what belongs halfway. That is frame interpolation. It smooths the motion.

It cannot fix identity. If the coat changed between the two real frames, the in-between just morphs one into the other. For a longer clip, a window slides along. Denoise frames one to sixteen.

Then nine to twenty four. The shared eight belong to both, so both must agree on them. Do that again and again and error compounds. Each chunk copies the last one's mistakes and adds its own.

Ten seconds in, the face is a different person. The other fix is optical flow. Measure where every pixel moved, drag the old frame forward, and compare. If the model disagrees, that is a penalty in training.

So, smooth comes from interpolation and a motion dial. Consistent comes from overlapping windows and flow. Two problems, wearing one name. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is an audio token? Let's try to understand. In a simple way. Sound is one number, over and over.

Air pressure, right now. Recording quality is forty four thousand of those every second. Guessing them one by one is hopeless. So first, a small network squeezes the sound.

It takes a slice of audio and describes it with a handful of numbers. Fifty slices a second. Then the trick. There is a fixed menu of sound chunks.

A codebook. Every slice snaps to the closest entry on that menu, and takes its number. And now music is a list of numbers. A row of them, in order.

That looks exactly like a sentence of tokens. So the language model machine just works. So the model does the only thing it knows. Guess the next token, add it, guess again.

Like writing a sentence, only each token is a slice of sound. But one chunk per slice sounds rough. So they stack them. The first token is a sketch.

The next codes what the first got wrong. Eight per slice. At the end, a decoder reads the tokens back. It rebuilds forty four thousand numbers a second, and you hear music.

The tokens were shorthand all along. One thing to be clear about. There are no notes inside. No chords, no sheet music, no instruments.

Just chunks of raw sound, numbered. So, music is written as words by squeezing sound into slices, snapping each one to a menu, and guessing the next number. Quick check now. One question is coming up.

Let's see if it clicked.

How does guiding music with text and melody work? Let's try to understand. In a simple way. A music model builds a track by guessing the next chunk of sound.

Your prompt goes through a text encoder, and the model reads it at every chunk. Cross attention. But text only gets you so far. Sad jazz trumpet pins the mood and the instrument.

Nothing about which notes. Ask twice, get two tunes. So there is a second control. Hand it a melody.

Even you humming. But you cannot feed the audio straight in. The model would just copy it. So the melody gets crushed first.

Every moment of it is sorted into twelve buckets, one for each pitch. How much of each is ringing. Then nearly all of it is thrown away. Only the loudest bucket at each moment lives.

The rest go to zero. That thin trace is a chromagram. Now both ride along. The text says what it should sound like.

The chroma says which pitch, at which moment. The model still just guesses the next chunk. And the thinness is the point. A pitch bucket holds no instrument, no recording.

So the tune survives, the sound does not. Hum it, get back an orchestra. One neat trick. Nobody labeled melodies for every song.

They pulled the chroma out of the very track the model was predicting. Free labels. So, text sets the sound, and a thin trace of pitch sets the tune. Two controls, and the model leans on both.

Quick check now. One question is coming up. Let's see if it clicked.

How does separating instruments in a generated track work? Let's try to understand. In a simple way. The music model finishes, and you get one wave.

Drums, bass, a voice, all summed into one number at every instant. Five plus three is eight. From eight you cannot tell. So nothing can be sliced apart.

A second model goes in after. It cannot cut the drums out. It can only guess them back. First the wave is laid out as a picture.

Time across, pitch up the side, brightness for loudness. A spectrogram. Now it is an image problem. The network paints a mask over it.

For every cell, a number from zero to one. How much of this is drums. Multiply, and the drums fall out. One mask per stem.

Drums, bass, vocals, everything else. At each cell they add to one. Every scrap is handed to somebody. Nothing is invented.

How did it learn? Studios keep the real stems. Add them up yourself and you already know the answer. Millions of times.

Undo my addition. It is a guess, so it leaks. A cymbal and a singer's hiss sit in the same cells. Both stems get a share.

That is the bleed. But a generated track has a shortcut. Build the model to emit several streams at once, one per instrument, and never sum them. Nothing to undo.

So, the mix threw the split away. A separator paints a mask per instrument and shares out every cell. Or the model never mixes at all. Quick check now.

One question is coming up. Let's see if it clicked.