Vision-language-action models

How does turning camera pixels into image patches work? Let's try to understand. In a simple way. A robot's camera captures a frame many times every second.

But to the model, that frame is not really a picture. It is just a giant grid of pixels. Millions of tiny numbers. The model cannot read millions of pixels one at a time.

That sequence would be far too long to handle. So we need a way to group them into bigger chunks. So we slice the image into a neat grid of small squares. Picture a checkerboard laid right over the photo.

Each square is one patch, maybe sixteen pixels wide. Every patch becomes one token. One single item in the model's input sequence. A whole camera frame shrinks from millions of pixels down to just a few hundred patches.

Inside each patch, the pixels get flattened and passed through a small layer. That turns the little square into one neat vector of numbers, ready for the model. Then we line the patches up in reading order. Left to right, top to bottom.

Now the image is a sequence, exactly like a sentence made of words. Why patches instead of raw pixels? Fewer items means attention can actually handle it. And each patch still holds a piece of the scene.

An edge, a corner, part of an object. So, camera pixels go in, and a grid of patch tokens comes out. The robot's view is now a sequence the model can finally read. Quick check now.

One question is coming up. Let's see if it clicked.

How does encoding a language instruction into tokens work? Let's try to understand. In a simple way. A robot gets its command as plain words, like, put the red block in the bowl.

But the model is just math. It cannot read letters. So the instruction is chopped into small pieces called tokens. The same trick a language model uses.

A token is a whole word, or a chunk of one. Every token matches an entry on a fixed list, the vocabulary. Each entry carries a number, its ID. So the words become a short list of IDs.

But an ID is just a name tag. It carries no meaning. So each ID looks up a learned vector in an embedding table. That vector captures the meaning.

Now the whole instruction is a short sequence of these vectors, one per token, in order. This sequence is what the model reads. Order matters. Block on plate is not plate on block.

So a position signal is added to each vector, marking where it sits in the line. Where do these vectors come from? Usually they are borrowed from a language model that already read the internet. So words carry meaning from the first step.

Here is the clever part. These word vectors have the same shape as the image patches. Same size, same space. That is what lets vision and language join later.

So, split the words, look up each ID, fetch its vector, add position. The command is now a row of tokens the model can think with. Quick check now. One question is coming up.

Let's see if it clicked.

How does merging image and text tokens work? Let's try to understand. In a simple way. By now we have two separate streams.

From the camera, a row of image patch tokens. From the instruction, a row of word tokens. Two piles, side by side. But the transformer wants one thing.

A single sequence of tokens. It does not have two doors, one for pictures and one for words. So both streams must become one line. For that to work, every token must be the same length.

But image tokens and word tokens often come out different sizes. So first we make their shapes match. A small layer, called a projection, resizes the image tokens. It stretches or squeezes each one until it is exactly as wide as a word token.

Now they share one space. Then we concatenate them. That just means lining them up end to end. The image tokens first, then the word tokens right after.

One long row, nose to tail. But now the model could lose track of which is which. So each token also gets a small type tag. A marker that says, I am vision, or I am language.

And just like words in a sentence, every token gets a position number too. That records where it sits in the merged line, so the order can never get scrambled. So, resize the image tokens, line them up with the words, tag each one, add positions. The two streams are now one sequence the transformer can read as a whole.

Quick check now. One question is coming up. Let's see if it clicked.

How does attention mixing vision with language work? Let's try to understand. In a simple way. We have one row now.

Image patch tokens, then word tokens, side by side. But they have not talked yet. Each token still only knows itself. Attention fixes that.

It lets every token look at every other token in the row. A word can look at patches. A patch can look at words. For that, each token emits three vectors.

A query, what am I looking for. A key, what I contain. And a value, what I pass on. The word block sends its query to every key in the row.

Each pair gets a score. Patches that look like a block score high. Empty floor scores low. Those scores get squashed into weights that add up to one hundred percent.

Most of the attention lands on a few patches. The rest gets almost nothing. Now the mixing. The word takes a weighted blend of the values it attended to.

Its vector comes back carrying the pixels of the block it found. That is grounding. The word red block is no longer just a word. It is bonded to a spot in the picture.

And patches attend to words the same way. This happens many times over. Several heads look for different things, color, shape, position. And every layer repeats it, so the blend gets richer.

So, query meets key, scores become weights, values get blended. Vision and language stop being two piles and become one shared understanding. Quick check now. One question is coming up.

Let's see if it clicked.

How does turning thoughts into action tokens work? Let's try to understand. In a simple way. By now, attention has mixed the picture with the words.

At the top of the stack sits one vector. The model's thought. Everything it understood, squeezed into numbers. But a robot cannot run a vector.

So something must turn that thought into an action. And here is the surprise. The model does it the same way it writes text. In a language model, the thought hits one last layer.

The output head. A single matrix that scores every entry in the vocabulary. Fifty thousand scores, one per token. Now the trick.

A slice of that vocabulary gets reserved for actions. Most numbers still spell words. But a block of them now stand for actions instead. Same menu, new section.

Then the usual step. Softmax turns those scores into probabilities, and the model picks one. If the winner lands in the action slice, out comes an action token. But one token is not a whole move.

So the model feeds it back in and predicts another. Then another. Just like writing a sentence, one word at a time. Why reuse the text head at all?

Because the model already knows how to pick tokens. All that web pretraining carries over. No new output layer. Just new words in the menu.

So, the thought hits the output head, softmax picks a token, and some tokens are actions now. Chain a few together and the model has written a move, not a sentence. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is an action token? Let's try to understand. In a simple way. Last time, a token came out of the model.

But what does it stand for? Start from the move itself. One step of a robot arm is just seven numbers. Three say how far to slide, along x, y and z.

Three say how far to twist. The last opens or closes the gripper. And these are changes, not places. Move two centimetres right from wherever you are.

The same token works anywhere in the room. Now the mismatch. Those seven numbers are smooth. They can be anything.

But a token is one entry from a fixed menu, which cannot hold every number. So we chop. Take one number's range, minus ten to plus ten centimetres. Slice it into two hundred and fifty six bins.

Every bin gets a token. So here is the answer. An action token is a bin number. It carries no value.

It points at a slot, and the slot has the value written on it. And the same token means different things depending on where it sits. First token, the x bin. Fourth, a twist.

Seventh, the gripper. Order carries the meaning. Chopping costs something. The arm can only land on bin edges, never between them.

Wider bins, coarser moves. More bins, a longer menu. Two fifty six is the usual compromise. So, an action token is one bin of one number of one small nudge.

Seven of them, in order, make a whole step. Quick check now. One question is coming up. Let's see if it clicked.

How does turning action tokens into motor commands work? Let's try to understand. In a simple way. Out of the model comes a row of action tokens.

Each is just a bin number, so the first job is a lookup. Bin one forty two means two centimetres. Do that for all seven and the numbers come back in order. Slide this much, twist this much, close the gripper.

One step in space. But no motor understands that. The arm has joints, and each joint has an angle. A step in space is a wish.

Joint angles are the command. So the model hands off. A separate low level controller sits underneath. Its job: turn where the hand should go into what each joint must do.

That job has a name. Inverse kinematics. Forward is easy: given the angles, where is the hand? Inverse runs it backwards, solving for the angles that reach the target.

There is also a speed gap. The model thinks a few times a second. Motors want a command hundreds of times a second. The controller smooths between them.

That is also why the model predicts a chunk: not one step, but the next several. The arm runs the chunk while the model thinks. And the controller is the last guard. Joint limits, speed caps, a box the hand may not leave.

A bad token asks. The controller refuses. So, tokens become numbers, numbers become a step in space, and a controller turns that step into joint angles. The model never touches a motor.

Quick check now. One question is coming up. Let's see if it clicked.

How does imitation learning work? Let's try to understand. In a simple way. The policy is the model that turns a picture and an instruction into action tokens.

But its weights start random. It flails around. Something has to teach it. So we show it.

A person drives the real robot with a controller, doing the task properly, again and again. Everything gets recorded. Then we chop that recording into moments. Each moment is a pair.

What the robot saw and was told, and the move the human made. That human move is the answer key. It becomes action tokens, and those tokens are the labels. No score, no reward.

Just the right answer, sitting there. Training is a quiz. Feed in the picture and the words, let the model predict its tokens, then compare them to the human's. Cross entropy turns the gap into one number.

Then backpropagation. Every weight gets nudged so the human's token would score a little higher next time. Millions of moments, millions of tiny nudges. That is the whole recipe.

It is called behavior cloning. No trial, no error, no reward. Just next token prediction, where the next token is a move. But copying has a ceiling.

The demos only show the task going well, so the policy never learns to recover. One wobble, and it is somewhere no demo covered. Guessing again. So, imitation learning is a human doing the task, chopped into pairs, with their move as the label.

Copy the teacher, one token at a time. Quick check now. One question is coming up. Let's see if it clicked.

How does learning from rewards and punishments work? Let's try to understand. In a simple way. Copying a human has a ceiling.

We let the robot try the task itself. Nobody drives it. Nobody says which move was right. All it gets is one number at the end.

One if the cup ended up lifted. Zero if not. That number is the reward. The robot acts alone, start to finish.

That whole run is one attempt. Fifty action tokens went out, one score came back. Now the hard part. One number for fifty moves.

Which move earned it? Which one spoiled it? Nothing tells you. That puzzle is called credit assignment.

The trick is to stop judging single moves. Judge the whole attempt. Scored high, nudge every token in it up. Scored low, nudge them all down.

That is a policy gradient. For that to work, the robot cannot always pick its top token. It samples, so it wanders a little. That is exploration.

Without variety, nothing new to compare. Notice, the reward never mentions a human. It only asks if the cup went up. So the robot can recover from a wobble, because recovering scored better.

It can beat its teacher. The catch is that reward is sparse. One number after a minute of moving. So it takes thousands of attempts.

Hand out small rewards along the way. That is reward shaping. So, rewards replace the answer key. Try, score the attempt, push the whole thing up or down.

Learning by consequences, not by copying. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the robot data bottleneck? Let's try to understand. In a simple way. Language models had it easy.

Trillions of words were already on the internet, written by other people. Nobody had to make that data. You download it. There is no internet of robot moves.

Nobody uploaded a million clips of an arm gripping a cup with the motor numbers attached. That data does not exist. So you make it yourself. A person teleoperates a real arm, one attempt at a time.

An hour of demos costs an hour. You cannot fast forward the real world. Between attempts, someone puts the cup back. Every single time.

The robot overheats. A cable snaps. Half the day is repair and reset. Compare the piles.

Text models train on trillions of words. The biggest shared robot dataset is about a million episodes. That gap is several zeros wide. Quantity alone is not enough.

A thousand demos of one cup on one table teach one cup on one table. Move the lamp and it fumbles. Variety multiplies the hours. And you cannot simply pool everyone's data.

My arm has seven joints, yours has six. The same action token means a different motion. That is the embodiment gap. So the bottleneck is not the model.

It is the data. That is why the field pools across robot types, and leans on simulation. So, robot data has to be lived, one real hour at a time. No archive, no shortcut.

That scarcity is the whole problem. Quick check now. One question is coming up. Let's see if it clicked.

How does training in simulation work? Let's try to understand. In a simple way. Real hours are the problem.

So you build the robot a video game. A physics engine with a table, a cup, and a copy of the arm. Inside it, time is yours. Run the physics faster than real.

Run two thousand copies at once. A year of practice finishes overnight. And nothing breaks. The cup resets itself instantly.

The simulator knows where every object sits, so labels and rewards come free. You train a brilliant policy, load it onto the real arm, and it misses the cup. The simulator was not reality. That is the reality gap.

The gap hides in small numbers. Real friction. The true weight of the cup. Camera colors.

And the milliseconds a real motor takes to answer. So the policy learned the simulator, not the task. It leaned on the exact friction it practiced against. Reality differs, and the grip slips.

The fix is domain randomization. Every single run, shuffle the numbers. New friction, new weight, new lighting, new table color. Never the same world twice.

Now no single physics can be memorized. The policy must find the move that works across all of them. Real friction becomes one more variation. Simulation does not replace real robots.

It buys the millions of cheap hours. A small batch of real demonstrations tunes the last mile. So, train where hours are free, then cross the reality gap by never letting the simulator be the same world twice. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is an object affordance? Let's try to understand. In a simple way. The model can see a mug and name it.

That does not help the hand. A name says nothing about where your fingers go. An affordance is what an object lets you do, and where. The handle affords a grip.

The rim affords a pour. A place, not the whole mug. Nobody hand labels this. No one draws a box that says, grip here.

It falls out of the demos. Every time, the fingers closed at the handle. So the vision encoder marks the patch. Not a label on the mug.

A heat map over the image. Bright at the handle, dark at the hot side. Web pretraining does most of the work. The model already knows handle and rim and blade from internet pictures.

Robot data only attaches an action to a word it already had. And it sticks to the shape, not the name. A mug it has never seen still has a handle shaped thing. Grab there.

Affordances carry over. Plain human video helps too. Watching hands touch things teaches contact points, with no motor numbers at all. Where a hand lands is a label.

One object holds many affordances. To cut, you grip the knife handle. To pass it over, you grip the blade. The instruction picks which one wakes up.

So, an affordance is an action attached to a place. Learned from where hands land, stored on the shape. That is how a robot knows where to grab. Quick check now.

One question is coming up. Let's see if it clicked.

How does breaking a long command into subtasks work? Let's try to understand. In a simple way. Say you tell the robot, clear the table, then wipe it down.

That is a minute of work, dozens of moves. The policy only emits about a second of motion. That is the gap. The policy predicts the next chunk from what the camera sees now.

Nothing in it holds a goal a minute away. So the system splits in two. A high level planner, usually a language model. And the low level policy that moves.

Steps up top, motions down below. The planner reads the long command and writes short steps. Pick up the cup. Drop it in the bin.

Wipe the table. Order is part of the plan. How do the two halves talk? In plain language.

The planner hands down one sentence at a time, the kind the policy already takes. That is the trick. The demos were short clips with short instructions. Pick up the sponge is in the data.

Clear the table never was. So the planner gives step one, watches until it is done, then gives step two. The policy never sees more than the step it is on. Step size matters.

Too big, and the policy is guessing a whole minute again. Too small, and you are writing motions yourself. One step, one skill. So, a long command becomes short named steps, handed down one at a time.

The planner holds the goal. The policy holds the skill. Quick check now. One question is coming up.

Let's see if it clicked.

How does thinking before acting work? Let's try to understand. In a simple way. Normally the camera view and the instruction go in, and action tokens come straight out.

One pass, nowhere to work anything out. That is a reflex. That is fine for, pick up the red block. Now try, hand me the thing I cut with.

Which object is that? A reflex cannot work it out. So let it write first. Before any action token, the model generates plain words.

The thing I cut with is the knife. The knife is on the left. The thoughts are not a separate module. Same model, same vocabulary, same stream.

The words land in the context, so the action tokens attend back to them. That is embodied chain of thought. That is the gain. The action token is now conditioned on facts the model wrote itself.

Demos have no thoughts. Nobody narrated. So the traces get written afterwards. A large vision language model reads the frames and the recorded action, then writes the reasoning that explains it.

Thinking costs time. Fifty reasoning tokens before every move, on a robot that wants decisions many times a second, and the arm stalls. So do not think every step. Reason once, then emit a whole chunk of actions from that thought.

Or split it. A slow model thinks, a fast one moves. So, thinking before acting is reasoning tokens in the stream, ahead of the action tokens. Written by a teacher model, paid for in time.

Quick check now. One question is coming up. Let's see if it clicked.

How does correcting mistakes with feedback work? Let's try to understand. In a simple way. The model plans the whole move, then runs it blind.

The gripper closes, the block slips out, the arm carries on holding nothing. That is an open loop. The fix sounds too simple. Look again.

Every few actions, take a fresh frame and feed it back in. Act, look, act, look. The loop is closed. Notice there is no error detector.

The new frame is the feedback. A shut gripper, a block on the table. The mistake is already in the pixels. The policy conditions on the frame in front of it.

So it never has to know it failed. Reaching again is simply the right next move. How often should it look? The chunk length is the loop period.

Long chunks run smooth but stay blind. Short ones react fast but jitter. There is a catch. A policy trained on flawless demos has never seen a dropped block.

Feedback shows the mess. Nothing tells it what to do. A human watches it run. The moment it goes wrong, they take the controls and steer it back.

That correction is logged, and now recovery is in the data. Feedback runs at two speeds. The inner loop lives on the robot, in milliseconds. The outer loop takes weeks.

It fails, a human corrects, the model retrains. So, a feedback loop is this. Look again, let the fresh frame carry the mistake, and train on the recoveries so it can use it. Quick check now.

One question is coming up. Let's see if it clicked.