Reinforcement learning, from zero

How does a machine learn to play a game when no one tells it what the right moves are? Most AI learns from labels. You show it a million pictures of cats. Each picture has a tag that says cat.

The AI just learns to match the picture to the tag. But imagine teaching a computer to ride a bike. There is no single correct steering angle for every second. You cannot label the perfect balance.

This is where reinforcement learning comes in. Instead of giving the AI a cheat sheet of right answers, you just give it a goal and a way to keep score. We call this score a reward. If the bike stays upright, it gets a plus one.

If it falls over, it gets a minus one. The AI tries random things at first. It falls over immediately. But eventually, by pure chance, it stays up a little longer and sees that positive score.

It connects that higher score to the actions it just took. It learns to repeat those actions. It is not copying a human. It is discovering what works.

The tricky part is that rewards are often delayed. In a chess game, the AI only gets a reward at the very end when it wins or loses. It has to look back at fifty moves and figure out which specific choices led to that final victory. That is the true puzzle of learning from rewards.

Let us see if you understand the core idea. Here is a quick check.

How does an agent see, act, and learn? Let's try to understand. In a simple way. Every reinforcement learning setup has just two characters.

The agent, the decision maker. And the environment, the world it lives in. It starts when the world hands the agent a state. A state is a snapshot of the situation right now.

Where things are, what is around. This is the agent seeing. The agent looks at that state and picks one action. The rule it uses to choose is called its policy.

State in, policy fires, action out. The action changes the world. So the environment sends back two things. A fresh state, showing what happened.

And a reward, one number scoring how good the move was. Here is the learning. If the reward was good, the agent nudges its policy to favor that action. If it was bad, it leans away.

Slowly the policy improves. See, act, learn. That is one step, called a timestep. Then it happens again, and again.

Thousands of these tiny turns are how the agent trains. One full run, from the start until the game ends, is called an episode. Play an episode, reset, play another. Each one teaches it a little more.

Picture a game character. The state is the screen. The action is a button press. The reward is the score going up.

See, act, learn, until it plays well. So an agent sees a state, acts through its policy, and learns from the reward. That one loop is the heartbeat of every reinforcement learner. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the explore versus exploit dilemma? Let's try to understand. In a simple way. An agent has one aim.

Collect as much reward as it can. But at every step, it faces a hard choice. One option is to exploit. Pick the action you already know pays well.

Like ordering your favorite dish. A safe bet, with a good reward right now. The other option is to explore. Try something you have not tested.

A new dish on the menu. It might be worse. But it might be even better. Here is the tension.

Every turn, you must pick one or the other. Time spent exploring is time not exploiting. That is the dilemma. So why not always exploit?

Because your favorite might only be second best. Never trying anything new, you stay stuck on a good option and miss the great one. And why not always explore? Because constant guessing wastes reward.

You keep sampling bad options, never cashing in on what you already learned works. The answer is a mix. Exploit most of the time to earn steadily. But now and then, explore, to keep learning if a better choice exists.

And the mix shifts over time. Early on, you know almost nothing, so you explore a lot. Later, once you have learned, you explore less. So the dilemma is simple.

Use what you know, or search for something better. Good agents do both, leaning on exploit but never closing the door on explore. Quick check now. One question is coming up.

Let's see if it clicked.

How does solving the best slot machine work? Let's try to understand. In a simple way. Picture a row of slot machines.

Each pays out at its own hidden rate. One is secretly the best, but you do not know which. This is the multi armed bandit. You pull one lever and get a reward, a win or nothing.

Here is the catch. You only learn about the machine you pulled. The rest stay a mystery. You have a limited number of pulls.

Your goal is to collect the most reward overall. So you must find the best machine, and find it fast. How do you judge a machine? Simple.

Keep a running average of what it has paid you. That average is your best guess at its true rate. You do not restart the math each time. After every pull, you nudge the average toward the new reward.

The step shrinks as pulls pile up. The more you pull a machine, the more that average settles onto its true rate. A few pulls are noisy. Many pulls tell the truth.

So you lean toward the machine with the highest average. But early on, those numbers are shaky. A good machine can look bad after two unlucky pulls. Every pull you spend on a weaker machine is a small loss, called regret.

Solving the bandit means keeping total regret as low as you can. So, a row of hidden payout rates. Average what each one pays, trust those averages more as they grow, and chase the highest. Quick check now.

One question is coming up. Let's see if it clicked.

How does a random guess work? Let's try to understand. In a simple way. Picture an agent facing a few choices.

Each time, it picks the one that looks best so far. Always the current favorite. That habit is called greedy. The trouble is, best so far can be wrong.

The first option that paid off becomes the favorite. And the agent stops checking the others. It gets stuck. Maybe one of the ignored choices is actually the best of all.

But the agent never tried it enough to know. Its true value stays hidden. Here is the fix. Every so often, ignore what you know.

Roll a die, and pick an action purely at random. A blind guess, on purpose. This trick has a name. Epsilon greedy.

Epsilon is a small number, the chance of guessing. Most of the time you exploit the favorite. A small slice, you explore. Why does a random guess help?

Because sometimes it lands on the option you undervalued. One good surprise, and the agent updates. Now it knows something new. The size of epsilon matters.

Too small, and the agent stays stuck in old habits. Too big, and it guesses so much it ignores what it already learned. A common trick is to start curious and calm down. Big epsilon early, when it knows nothing.

Then shrink it, as the estimates get trustworthy. So the power of a random guess is escape. It breaks the agent out of a comfortable rut, and lets a better option finally prove itself. Quick check now.

One question is coming up. Let's see if it clicked.

How does mapping actions to future rewards work? Let's try to understand. In a simple way. In the real world, an action does two things.

It hands you a reward now. And it moves you to a new situation, where more choices wait. So judging an action by its reward right now can fool you. The move with the fattest instant payoff might lead into a dead end.

Because every action starts a chain. You act, you land somewhere, you act again. Rewards keep arriving, long after the first one. The honest way to score an action is to add up every reward that follows it.

That running total has a name. It is called the return. Now pin that total onto the action itself. This move tends to lead to this much future reward.

That number is the action's value. The agent does not know these values at first. It estimates them from experience, and sharpens each guess as it sees how things play out. Line them up in a table.

For every situation, a value for every action. That table is the map from actions to their future rewards. Now choosing is easy. Compare the values, not the instant rewards.

This lets the agent give up a small prize now, to reach a bigger one later. So, an action is worth the whole future it unlocks. Sum the rewards that follow, store that value, and the agent can think past the next step. Quick check now.

One question is coming up. Let's see if it clicked.

How do future rewards shape today's choice? Let's try to understand. In a simple way. An agent makes a move.

It gets a small reward right now. But today's choice is not just about this instant. It can set up a much bigger reward later. So instead of only the next reward, the agent adds up every reward that follows.

That whole stream, summed together, is called the return. But a reward far away is worth less than one right now. It is less certain, and comes later. So each step ahead, we shrink its value.

That shrinking uses one number, gamma, the discount factor. It sits between zero and one. Reward now counts fully. The next step times gamma.

Two steps out, gamma times gamma. This dial sets the agent's horizon. Gamma near zero makes it short-sighted, chasing only the reward in front of it. Gamma near one makes it far-sighted, weighing the distant future.

So the agent chases the discounted return, immediate reward plus all the shrunken future ones. That sum guides today's decision. The payoff. A tiny treat now, or a larger reward two steps ahead.

A short-sighted agent grabs the treat. A far-sighted one waits, because the discounted future wins. Why shrink at all? Without it, an endless task could grow to infinity.

Discounting keeps the total finite, and trusts the near future more than the far. So a choice today is judged by more than the next reward. It is judged by the return: future payoffs summed and discounted by gamma. Quick check now.

One question is coming up. Let's see if it clicked.

How does an agent learn from its surprises? Let's try to understand. In a simple way. Before it acts, an agent carries a guess.

A prediction of how much reward this move should bring. That is what it expects. Then it takes the step, and reality answers. A real reward lands.

And it sees the new spot it moved into. Now compare the two. What it expected, against what actually happened. That gap has a name.

It is the prediction error, the agent's surprise. This surprise points a direction. Better than expected, and the guess gets nudged up. Worse than expected, and it gets nudged down.

What does it compare against? The reward it just got, plus its own guess of what comes next. That total, minus the old guess, is the temporal difference error. It does not overwrite the old guess.

It steps a small fraction toward the target. That fraction is the learning rate. New guess equals old guess, plus a slice of the surprise. And when the guess already matched what happened?

The surprise is zero, so nothing changes. Learning only touches the guesses that were wrong. Here is the clever part. The agent never waits for the game to finish.

Each step corrects the last guess using the next one. Learning from a guess, with a guess. That trick is called bootstrapping. So an agent learns by being surprised.

Predict, act, measure the gap, then nudge toward what really happened. Step by step, the surprises fade, and the guesses come true. Quick check now. One question is coming up.

Let's see if it clicked.

How does an agent learn when the world is too big for a table? Let's try to understand. In a simple way. Remember the value table.

One row per situation, a value per action. To choose, look up the row. It works while situations stay few. But picture a real world.

Every camera frame, every board position, is its own situation. The count explodes past every atom in the universe. Two walls hit at once. No computer could store a row for each.

And the agent could never visit them all, so most rows stay blank. So we stop naming each situation. We describe it with a few features, numbers that capture its look. Similar situations share features.

Now swap the giant table for a function. It reads the features and computes a value on the spot. This is called function approximation. Here is the payoff.

The function generalizes. Learn a value in one situation, and every situation with similar features shifts too. What is that function, usually? A neural network.

Features in, a value out. That pairing is deep reinforcement learning. Learning changes shape too. Instead of writing one cell, the agent tunes the network's shared weights.

One update ripples across related situations. There is a catch. Values are estimates now, not exact, and updates can bleed into situations you never chose. It is the only way to scale.

So when the world is too big for a table, describe situations by features and let a function value them. It covers what no table could. Quick check now. One question is coming up.

Let's see if it clicked.

How does learning a policy instead of values work? Let's try to understand. In a simple way. So far, the agent learned values.

A number for how good each action is. To act, it scored every action, then picked the highest. But notice something. The agent never wanted those numbers.

It only needs to know what to do. The values were a detour. So here is the new idea. Skip the values completely.

Learn the choice itself, straight from the situation. That direct rule is the policy. A policy is a map. It reads the situation and hands back an action.

Value methods built it the long way. Now we learn it head on. Usually a policy does not name one action. It gives each action a probability.

A big share to the moves it favors, a sliver to the rest. Why probabilities, not one pick? Because sometimes the best play is to stay unpredictable. Think rock paper scissors.

A mix beats any fixed move. And some actions are continuous. A steering angle, an amount of force. You cannot score every one and pick the max.

A policy just outputs the number. What is the policy, really? A network of weights. Features go in, action probabilities come out.

Tuning them shifts which actions it prefers. So there are two routes to one goal. Value methods go through numbers first. A policy jumps straight from situation to action.

So instead of rating every action, a policy learns the action itself. Situation in, move out. No number in between. Quick check now.

One question is coming up. Let's see if it clicked.

How does getting rewarded for good instincts work? Let's try to understand. In a simple way. Remember, a policy is a set of instincts.

In each situation it leans toward some actions and away from others. But how do those leanings get better? Here is the catch. Nobody tells the agent the right move.

There is no answer key saying this action was correct. It only gets rewards. So it does the only thing it can. It plays a whole run, acting on its current instincts, and watches how much reward piles up by the end.

That total is the one signal it has. We call it the return. A good run scores high. A bad run scores low.

Now the key move. If the run went well, make every action you took more likely. If it went badly, make them less likely. That is the whole idea.

A good outcome rewards the instincts that produced it. Next time, those hunches fire a little stronger. This nudge has a name. The policy gradient.

The method built on it is called reinforce. Reward flows backward and reshapes the leanings. And the push scales with the reward. A big win shoves the probabilities hard.

A small one barely moves them. But watch the subtlety. It cannot tell which single action deserved the credit. It nudges all of them, and trusts many runs to sort the true winners out.

So the agent acts, measures the outcome, and reinforces what led to reward. Good instincts, paid forward. Quick check now. One question is coming up.

Let's see if it clicked.

How does the actor critic method work? Let's try to understand. In a simple way. Last time, the agent learned by playing a whole run and reinforcing the actions that led to reward.

But that final score is noisy, and it waits until the very end. So we split the job in two. The first part is the actor. It is the policy that performs.

In each state, it picks the action. The second part is the critic. It does not choose anything. It only judges.

It watches the state and estimates how good things look from here. That estimate predicts the return you should expect from this spot. Think of it as a baseline. Business as usual.

Now the trick. After an action, compare what actually happened to what the critic expected. Did it turn out better than the baseline, or worse? That gap is the signal.

If the action beat the expectation, the actor makes it more likely. If it fell short, less likely. And the critic can judge every single step. So the actor gets feedback right away, instead of waiting for the whole run to finish.

Subtracting the critic's baseline also cancels much of the noise. Learning becomes steadier, needing fewer runs to improve. Meanwhile the critic is learning too. It nudges its own predictions toward what really occurred.

Actor and critic sharpen each other, together. So one part acts, the other part judges, and the judgment tells the actor what to reinforce. A performer, and a coach. Quick check now.

One question is coming up. Let's see if it clicked.

How does teaching AI with human preferences work? Let's try to understand. In a simple way. A reward is just a number.

In a game it's obvious: points, or a win. But was this answer helpful and well written? There is no number for that in the world. You could ask a person to score it, zero to ten.

But absolute scores are shaky. My seven is your four, and scales drift. So flip the question. Show the person two answers to the same prompt, A and B.

Then ask one thing. Which one is better? People are good at that. Now do this thousands of times, across many prompts.

Each pick is a sliver of human taste. Soon, a big pile of comparisons. A beats B, D beats C. That pile is a new kind of training signal.

Not a correct answer to copy. Just a preference. This beats that. Quality judged, not spelled out.

And that is why comparisons win. Choosing between two is easier and steadier than pinning down a number. Two raters drift apart. Two choosers agree.

Then the model is trained to make preferred answers more likely, and rejected ones less likely. That pile of choices pulls its behavior toward what people want. And this captures what no rule could write down. Politeness.

Helpfulness. Whether an answer just feels right. Fuzzy human values, taught by example choices. So the whole recipe has a name.

Reinforcement learning from human feedback, or RLHF. Learn from what people prefer, not from a fixed answer key. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a reward model? Let's try to understand. In a simple way. Last lesson, people compared answers by hand.

But training needs a reward for millions of attempts. No human could sit and score them all. So here is the move. Train a second model whose only job is to imitate that human taste.

Call it the reward model. The reward model reads an answer and puts out a single number. A score. How much a person would probably like this reply.

How does it learn? From that pile of human comparisons. For every pair, nudge its weights so the preferred answer earns a higher score than the rejected one. It never needs the true value of an answer.

It only has to rank the pair the right way. Winner above loser, again and again. After enough pairs, something useful appears. The reward model scores brand new answers no human ever saw.

It learned the pattern of taste, not the exact examples. Now it runs on its own. The reward model scores answer after answer, instantly, standing in for the person. That score becomes the reward the agent chases.

But remember what it is. A proxy. A learned guess at human taste, not the real thing. Push too hard and the agent can game its blind spots.

So a reward model turns a pile of human choices into an automatic judge. One number, standing in for what people like. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a trust region? Let's try to understand. In a simple way. An agent just learned which way is better.

A reward pointed uphill. Now it updates its policy that way. But one question decides everything. How big a step?

Here is the catch. The agent learned from its own recent tries. That data only describes how it acts right now. It is a snapshot of one small neighborhood.

Take one giant step, and the policy leaps somewhere new. Somewhere that snapshot never covered. The direction looked good nearby, but far away it was only a guess. And in reinforcement learning, there is no undo.

A bad leap can wreck the policy, and it feeds its own next round. Fall off the cliff, and it may never climb back. So the fix is simple. Take a small, careful step.

Stay inside the region where that recent data still holds. Learners call it the trust region. Then stop and look again. Collect fresh tries from the updated policy.

Re-aim, and step once more. Many small steps, never one reckless leap. Modern methods build this in. A popular one, called PPO, clips each update so the new policy can never drift too far from the old one.

Too big, and it overshoots and collapses. Too tiny, and it barely learns. The step size is the dial that keeps learning both fast and safe. So, small steps keep the policy close to the data it trusts.

One giant leap can throw all that progress away. Quick check now. One question is coming up. Let's see if it clicked.