World models and simulation

How do agents build their own realities? Let's try to understand. In a simple way. An agent lives inside some world.

A game, a road, a warehouse. Normally it learns by acting, watching what happens, then adjusting and trying again. But acting for real is slow. It can be expensive.

And it can be dangerous. You cannot crash a real car a thousand times just to learn to drive. So the agent does something clever. From its own experience, it builds a small copy of the world, inside itself, that it can run whenever it likes.

This inner copy has a name. A world model. A learned predictor that guesses what happens next whenever the agent takes an action. Now the agent can ask, what if I do this?

And it can answer inside its own head, without ever touching the real world. That is why we say its own reality. The model is built only from what this agent has seen. So it is personal, and never quite perfect.

It is not the real world. It is a rough reconstruction, just good enough to think with. A private map of how things work. And that is the whole point.

With a world living in its head, the agent can think before it acts. Imagination, in place of slow trial and error. So, an agent builds a world model. A learned inner reality.

A safe place to imagine what its actions would do, before doing them. Quick check now. One question is coming up. Let's see if it clicked.

How does turning pixels into a compressed memory work? Let's try to understand. In a simple way. An agent looking at a game sees raw pixels.

Just a huge grid of brightness numbers. One small frame can be thousands of values. Far too much to think with directly. So the first step is to squeeze that image down.

We shrink the thousands of pixels into a short list of numbers. A tiny summary of the scene. That short list is called the latent vector. Maybe just thirty two numbers, standing in for the whole picture.

This is the compressed memory. What does the squeezing? A small neural network called the encoder. It passes the image through layers that shrink it, step by step, into that short code.

But how do we know the code kept the important parts? A second network, the decoder, tries to rebuild the full picture from the code alone. Encoder and decoder train together. This pair is called an autoencoder.

Squeeze, then rebuild. When the rebuilt image matches the original, the code must be good. To rebuild well, the code learns to keep what matters. The walls, the road, the enemy ahead.

It happily drops the exact value of every pixel. From now on, the agent thinks in this compact code, not in pixels. It is the memory the rest of the world model actually uses. So, an encoder squeezes each frame into a tiny latent vector.

A decoder proves nothing vital was lost. That small code is the compressed memory. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a latent space dream? Let's try to understand. In a simple way. First, remember the setup.

The agent does not store the world as pictures. It squeezes each moment into a tiny code. A short list of numbers, called a latent. A dream begins when the agent stops looking at the real world.

Instead it takes the current code and imagines the next one, all on its own. Here is the key. This all happens in latent space. The agent dreams in codes, never drawing a single pixel.

No images are made at all. Why call it a dream? Because none of it is real. The scene is generated from inside the model, like a dream in your sleep.

The dream rolls forward in a loop. Take a code, pick an action, predict the next code. Then feed that guess back in as the new now. Because a code is just a handful of numbers, each dreamed step is tiny and fast.

The agent can dream thousands of moments almost instantly. If we are curious, we can decode a code back into a picture, just to peek at the dream. But the agent itself never needs those pictures. And nothing real is touched.

No game is played, no car is moved. The whole dream is a private film running inside the agent's mind. So, a latent space dream is imagined experience, made of codes, not pixels. The world model plays it forward, cheap and safe, all inside the agent's head.

Quick check now. One question is coming up. Let's see if it clicked.

How does an agent predict the future one action at a time? Let's try to understand. In a simple way. Start with what the agent already holds.

The current moment, squeezed into a tiny code. A short list of numbers that stands for the state right now. To guess the future, you need one more thing. The action the agent is about to take.

Steer left, press a button, or step forward. Now feed both into the world model. The current code, and the chosen action. Out comes a guess at the very next code.

This predictor has a name. The dynamics model. It is a learned function that takes a state and an action, and returns the next state. Notice that it jumps just one step.

Not the whole future, only the very next moment. One action in, one next state out. The action really matters. From the same state, steering left and steering right predict two different next codes.

Your choice changes what comes next. Usually it guesses a little more. Alongside the next state, it also predicts the reward. How good or bad that step is likely to be.

This single step is the building block. Predict one moment, then feed the guess back in to picture many. But that chaining comes next. So, the world model predicts the future one step at a time.

Feed in a state and an action, and out comes the next state, with its reward. Quick check now. One question is coming up. Let's see if it clicked.

How do agents learn the rules of their world? Let's try to understand. In a simple way. The world model starts out knowing nothing.

It cannot guess what happens next, because no one handed it the rules of the game. So the agent simply acts and watches. Each moment, it records three things. The state it was in, the action it took, and the state that actually came next.

Do this again and again, and you pile up a long log of these little triples. A record of what really happened, straight from experience. Now training begins. Show the model a state and an action from the log, and let it guess the next state.

We already know the true answer, because it was recorded. Compare that guess to what really happened. The gap between them is the prediction error. A single number for how wrong the model was.

Then shrink that error. Nudge the model's weights a little, in the direction that makes the guess closer. This is plain gradient descent. Repeat over millions of recorded moments.

Slowly the errors shrink, and the guesses start to match reality. The model is soaking up the rules. Notice that no one wrote down any physics. Reality itself hands over the answer key, for free.

That is why we call this self supervised learning. So, the agent collects real transitions, predicts the next one, and shrinks the error. Do that enough, and the model learns the rules of its world. Quick check now.

One question is coming up. Let's see if it clicked.

How does chaining predictions into imagined futures work? Let's try to understand. In a simple way. Recall the single step.

Feed the world model a state and an action, and it guesses the very next state. One moment ahead, and no more. But one step barely helps. To really plan, an agent wants to see far ahead.

Many moments into the future, not just the next one. Here is the trick. Take that predicted next state, and treat it as if it were real. Feed it straight back into the same model as the new now.

Now add the next action, and predict again. Out comes the state after that. You have just taken a second step, without ever touching the real world. Keep looping.

Predict, feed back, predict, feed back. This chain of predictions is called a rollout. An imagined trajectory of states. How many steps you chain is the horizon.

Chain ten, and the agent pictures ten moments ahead. All inside its own head, with no real environment at all. Feed in a different sequence of actions, and you get a different imagined future. So the agent can compare plans, all before it lifts a finger.

Notice that the loop is closed. The model's own guesses become its next inputs. That is powerful, though small errors can quietly build with every step. So, chaining predictions turns a single step into a whole future.

Feed each guess back, add an action, and repeat, and the agent dreams many moves ahead. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the blur in an imagined future? Let's try to understand. In a simple way. Recall the imagined rollout.

Each step feeds its own guess back in, then predicts again. A chain, all inside the model's head. Here is the catch. Every prediction is a tiny bit off.

Even a well trained model is never perfect, so each guessed state carries a small error. Now watch feeding back. That slightly wrong state goes right back in, as if it were true. So the next step begins from a flawed input.

And it adds its own fresh error on top of the inherited one. Step after step, the errors stack up. This piling on is called compounding error. It gets worse.

The model only trained on real states. Its own drifted guesses look unfamiliar, off the training distribution, so it predicts them even worse. So the imagined trajectory slowly drifts from what reality would really do. The first few steps stay sharp, but later ones wander further and further off.

That growing uncertainty is the blur. Far into the imagined future, the prediction becomes a fog of maybes, not a crisp picture of what comes next. So agents lean on the near future, where the dream is still sharp. Keep the horizon short, and plan on predictions you can actually trust.

So, each guess is slightly off, feeding it back stacks the error, and the far future blurs. Trust the near steps, keep your dreams short. Quick check now. One question is coming up.

Let's see if it clicked.

How does searching for rewards inside a dream work? Let's try to understand. In a simple way. Recall the dream.

The agent can roll out imagined futures in its own head, without ever touching the real world. But a stream of imagined states says nothing about what is good. The agent needs to know which futures are worth chasing. So the world model predicts one more thing.

Alongside each next state, it guesses a reward, a single number saying how good that imagined moment is. Now roll a plan forward and add up those predicted rewards. That sum is the return, one score for the whole imagined trajectory. One plan is not enough.

So the agent dreams up many different action sequences, each a candidate plan it might follow from right now. It rolls every candidate out inside the dream and scores each by its total predicted reward. Good plans earn a high return, bad ones a low score. Then it simply picks the winner.

The plan with the highest imagined return. This search for the best plan is called planning, or model predictive control. But it does not commit to the whole plan. It takes just the first action for real, then looks again and plans afresh.

And all this searching costs nothing in the real world. Thousands of plans can be tried and discarded safely, all inside the model's imagination. So, the model predicts rewards too, you sum them into a return, try many plans, and pick the highest. Reward search, all inside a dream.

Quick check now. One question is coming up. Let's see if it clicked.

How does picking actions from imagined successes work? Let's try to understand. In a simple way. Recall that the agent can imagine a plan and score it by return, its total predicted reward.

But how should it choose which plans to even try? The lazy way is pure guessing. Sample action sequences at random, and most of them are junk. You waste nearly every imagined rollout on a bad plan.

So start smarter. Draw a whole batch of candidate plans from a broad distribution over actions, wide and unsure at first, covering many possibilities. Roll every candidate forward inside the world model, and score each one by its imagined return. Now you know which plans looked good.

Here is the key move. Keep only the top handful, the highest scoring plans. These winners are called the elites, your imagined successes. Then refit.

Build a fresh distribution centered on those elites, its mean and spread pulled toward what just worked. The search tightens around success. Now sample again from that better distribution, score, and keep the new elites. Repeat a few rounds.

This refine loop is the cross entropy method. After a handful of rounds the plans cluster on high return actions. Take the first action of the best one for real, then plan afresh. So, sample plans, imagine them, keep the successes, and refit toward them.

Round by round the agent learns to pick actions from its own imagined wins. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is acting with a map versus pure instinct? Let's try to understand. In a simple way. An agent with a world model still has to pick an action at every step.

And it turns out there are two very different ways to decide. The first way is pure instinct. A trained network, called a policy, takes the current state straight in, and an action pops straight out. No imagining at all.

This is like muscle memory. It is fast, cheap, and automatic. But it only knows the moves it was drilled on, and it never looks ahead. The second way is acting with a map.

Before it moves, the agent uses its world model to imagine what a few different actions would lead to. This one is flexible. It can plan toward almost any goal. But it is slow, because every single decision now costs a fresh burst of imagining.

Think of a daily commute you drive on autopilot, versus reading a map in a brand new city. One is instant. The other thinks before each turn. So it is a real trade.

Instinct is quick but rigid. The map adapts to anything, yet it burns compute at every step. The clever move is to use both. Lean on instinct for the routine, and unfold the map only when the situation is new, or the stakes are high.

Quick check now. One question is coming up. Let's see if it clicked.

How does learning faster with imagined practice work? Let's try to understand. In a simple way. Learning by doing is slow.

Every real trial takes time, costs energy, and a bad one can break things. Real experience is the bottleneck. But the agent already carries a world model. A learned simulator that, given a state and an action, predicts the next state and the reward.

So let it practice inside that model. Instead of acting for real, it dreams up transitions, imagined experience it can learn from. And imagination is cheap. No robot to move, no clock to wait on.

The agent can run thousands of practice steps in the time one real step would take. This gives a simple loop, often called Dyna. Take one small step in the real world. Store it.

Then pause and run many imagined updates from the model. The policy does not care where a step came from. Real transitions and imagined ones flow through the exact same update rule. The payoff is sample efficiency.

The agent reaches the same skill with far fewer real trials, so it learns much faster. Notice the trick. Imagined practice does not replace real experience. It stretches it.

Each real trial teaches far more once you also rehearse it in your head. So, a world model turns scarce real data into cheap practice. Act a little, imagine a lot, and learn faster from every trial. Quick check now.

One question is coming up. Let's see if it clicked.

How does training a policy on pure imagination work? Let's try to understand. In a simple way. Two pieces.

The policy is a reactive network: state in, action out. The world model is a learned simulator of the environment. Now the bold move. Do not train the policy in the real world at all.

Train it entirely inside the world model, on dreamed rollouts. Here is how a dream teaches. From a state, the policy acts and the model rolls the future forward. Sum the rewards into a return.

Higher return, better actions. And the world model is differentiable. So the reward gradient flows backward through the whole imagined trajectory, straight into the policy weights. The dream itself says which way to nudge.

But a dream can only run so far before it blurs. So a value network guesses the rewards beyond the horizon. Even a short dream sees the long game. Where does real experience go?

Only into the world model, to keep it accurate. It never touches the policy update. The policy learns purely from imagination. So the loop is simple.

Collect a little real data. Sharpen the model. Then train the policy for thousands of steps inside the dream. Repeat.

There is a catch. The policy is only as good as the dream. If the model has a blind spot, the policy exploits that fantasy, then stumbles for real. So, real trials feed the model, and the model feeds the policy.

Train on pure imagination, and one reactive network comes out ready to act. Quick check now. One question is coming up. Let's see if it clicked.

How does correcting a dream with real data work? Let's try to understand. In a simple way. First, the dream.

An agent carries a world model, a learned guess of how the world responds. It practices inside that guess, not reality. But a learned model is never perfect. As the policy improves, it wanders into corners the model never saw.

Errors pile up, and the dream drifts from reality. So the fix is to test the dream against truth. Let the agent take a real step. The dream predicted one next state.

Reality delivered another. The gap between what the model predicted and what really happened is the prediction error. That gap is the correction signal, pointing to where the dream is wrong. So the real transitions flow into the world model.

A gradient step shrinks the error, teaching the dream what really comes next. Real data fixes the model, not the policy. And the correction lands where it matters. The fresh data comes from the states the policy now visits, so the dream is patched exactly where the agent goes.

So the two worlds take turns. Dream up practice to improve the policy. Run it for real, collect fresh data, correct the model. Dream again.

Why bother? Without this, the policy learns to exploit the dream's mistakes, winning in fantasy but failing for real. Real data keeps the dream honest. So, real trials are the ground truth.

The gap between prediction and reality retrains the model, keeping the dream anchored to the world it acts in. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a broken world model? Let's try to understand. In a simple way. An agent's world model is trained only on states it has visited.

Inside that seen region, its predictions are trustworthy. It learned the pattern there. But push the agent into a truly new state, far from its training data, and the model is off the map. It has nothing to base a guess on.

Here is the danger. The model does not go blank. It still outputs a crisp, confident next state. But that prediction is invented.

It breaks quietly, no warning. And the policy makes it worse. Hunting for high reward, it often finds that reward hiding in corners the model never saw, straight off the map. So how do we catch a break?

Train several models on the same data. On familiar states they all agree. On a strange new state, their guesses scatter. That spread is the model's uncertainty.

It flags where the model is guessing, not knowing. Wide disagreement means, do not trust the dream here. So we add a guardrail. Penalize imagined rewards from uncertain states, or halt the rollout when the models disagree.

Keep the agent inside the region the model trusts. This principle is pessimism. When the model is unsure, assume the worst, not the best. It stops the policy chasing rewards that only exist in a broken dream.

So, a world model breaks off its training data, failing with false confidence. Measure that uncertainty, stay where it is trusted, and the dream stays safe. Quick check now. One question is coming up.

Let's see if it clicked.

How does a language model work as a world model? Let's try to understand. In a simple way. First, recall the shape of a world model.

It takes a state and an action, then predicts the next state. Just a learned next-step predictor. Now a language model. It reads the text so far and predicts the next token.

Read the past, guess what comes next. The very same shape. So line them up. The running text is the state.

The words you feed in are the action. And the next token is the predicted next state. Chain those predictions and you get a rollout in words. Drop a glass.

It imagines what comes next. It falls. It shatters. A future in language.

Why does it work? Pretraining on the whole internet packed in huge world knowledge. Cause and effect, rough physics, how people act. A broad simulator, for free.

But here is the catch. It models the world of text, not the world itself. With no grounding, it can invent a fluent next state that is simply false. The fix is grounding.

Feed real observations back as tokens, or add vision and other senses. Now the actual world steers it, not just what sounds right. Why care? You get a world model you can prompt in plain words.

Describe a plan, ask what happens next, read the imagined outcome. So, a language model is a next-token predictor, which makes it a world model over text. It gained world knowledge free, but needs grounding to stay true. Quick check now.

One question is coming up. Let's see if it clicked.