Vision transformers

What exactly is a CNN's tunnel vision? Let's try to understand. In a simple way. A CNN never sees a whole image at once.

It scans with a tiny filter, a window just three pixels by three. It slides across the picture, one step at a time. At each stop, the filter sees just that little patch and nothing else. Nine pixels of the image.

This small neighborhood is called its receptive field. So a single filter only ever connects pixels that sit close together. Put an ear in one corner and a tail in the far corner. Inside one layer, those two spots can never meet.

How does a CNN ever see the big picture? It stacks many layers. Each layer looks at the summaries below it, so the receptive field grows a little wider with depth. But that widening is slow.

For two far corners to finally influence each other, you may need dozens of layers stacked up. A global view arrives late. This is the tunnel vision. Early in the network, every neuron peers through a narrow tube, aware of its little neighborhood but blind to the rest of the scene.

This narrow focus is not all bad. It makes CNNs fast and sharp at spotting edges and textures. But understanding the whole scene at once does not come for free. So, a CNN has tunnel vision because its filters are small and local.

Each one sees a tiny patch, and a wide view only builds up slowly, layer by layer. Quick check now. One question is coming up. Let's see if it clicked.

How does treating an image like a sentence work? Let's try to understand. In a simple way. The transformer was invented for language.

And a sentence is just an ordered sequence of words, one after another. Read left to right, in a line. Its real trick is that it reads the whole sentence at once. Every word can look at every other word directly, even ones far apart.

But this machine only speaks one language. Sequences. Hand it a flat sentence and it is happy. Hand it a two dimensional grid of pixels, and it has no idea what to do.

So here is the bold idea. We make the image look like a sentence. Cut it into small pieces, and lay them out in a single line, read in order. Now the analogy clicks.

Each little piece plays the role of a word. A visual word. And the whole image plays the role of a sentence. Compare this to a convolutional network.

It slides a small window and only sees a local patch at a time. The transformer instead takes in the entire sequence together. And that is the payoff. Because the image is now one sequence, a piece in one corner can connect to a piece in the far corner right away.

Global view from the very first step. So, we treat an image like a sentence. Cut it into a line of pieces, let each piece act like a word, and a transformer can finally read the whole picture. Quick check now.

One question is coming up. Let's see if it clicked.

How do we actually feed a photograph into a transformer? Transformers were originally built for text. They read sentences one word at a time. But an image does not have words.

It just has a massive grid of colored pixels. You might think we should just feed those pixels in one by one. But a standard phone photo has twelve million pixels. Processing twelve million separate items at once would instantly crash a transformer.

The math simply gets too heavy. We need chunks that are bigger than a single pixel, but smaller than the whole image. So we have to chop the picture up. Imagine laying a piece of graph paper over your photograph.

You draw a rigid grid across the entire thing, dividing it into equal parts. A very common grid size uses squares that are exactly sixteen by sixteen pixels. We slice the image cleanly along these lines. Instead of millions of tiny dots, we now have a much smaller, manageable number of little squares.

We call these individual squares patches. What is inside a patch? It really depends on the picture. One square might show a dog ear.

Another might just be empty blue sky. We take these patches and arrange them left to right, top to bottom. It is exactly like reading words in a book. The photograph is no longer a single giant rectangle.

It is now a sequence of little square images lined up in a row. The transformer finally has something it can work with. Quick check. Which of these statements is actually true about the patches?

What exactly is unrolling a patch into a vector? Let's try to understand. In a simple way. Remember, we already chopped the image into small square patches.

But a patch is still a tiny two dimensional grid. Just rows and columns of pixels. A transformer does not read grids. It reads a flat list of numbers, laid out in a straight line.

So this shape is simply wrong. So we unroll the patch. We take the top row of pixels, then the next row, and lay them all end to end. One long line.

The order is fixed. Always left to right, then top to bottom. The same path for every single patch, so nothing ever gets scrambled. But here is a twist.

Each pixel is not one number. It carries three. A red value, a green value, and a blue value. So let's count.

A sixteen by sixteen patch holds two hundred and fifty six pixels. Times three colors. That is seven hundred and sixty eight numbers. Line those numbers up and you get a vector.

Seven hundred and sixty eight values long. That is the whole patch, written as one flat list. Notice we have not learned anything yet. These are just the raw pixel values, in a fixed order.

Nothing clever here, only reshaped. And every patch becomes a vector of the exact same length. So they stack neatly into a sequence the transformer can read. Unroll, and you are done.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is giving each patch its own meaning? Let's try to understand. In a simple way. So far, each patch is just a long list of raw color numbers.

Red, green, and blue values, lined up in order. Nothing more. But those raw numbers barely tell the model anything. A brightness here, a shade of blue there.

They do not say what the patch actually shows. So we multiply that raw vector by a grid of weights. This is a simple linear layer. Every output number mixes all the inputs together.

Out comes a brand new vector. We call it the patch embedding. It is the same patch, but now written as meaning, not raw color. Here is the key.

Those weights are learned during training. The model slowly tunes them so useful patterns get their own strong signals. And it is one single grid of weights, shared by every patch. The same projection runs on each one, so they all speak the same language.

This is exactly how language models treat words. Each word becomes a learned vector. Here, each patch becomes a visual word, a token with meaning. The embedding has a fixed length, the same for every patch.

The model picks that size, often several hundred numbers wide. So, raw pixels go in, a learned projection reshapes their meaning, and a rich embedding comes out. Now the transformer finally has real tokens to work with. Quick check now.

One question is coming up. Let's see if it clicked.

How does a patch know its location? Let's try to understand. In a simple way. By now our image is a line of patches, fed into the transformer all at once.

But something important got left behind. The attention layer reads all the patches together, as one big bag. On its own, it has no idea which patch sat where in the picture. So here is the strange part.

Shuffle the patches into a random order, and the transformer sees the very same thing. The layout simply does not reach it. But position clearly matters. Sky sits at the top, ground at the bottom.

An eye belongs above a nose. Where a patch lives is part of the meaning. The fix is simple. Give every slot its own little vector of numbers.

Position zero, position one, position two, and so on down the line. These position vectors are not fixed by hand. The model learns them during training, tuning each one until it captures where that spot lives. Now the clever step.

We add the position vector straight onto the patch vector. Not glued to the side, added on top, number by number. So each patch now carries two things at once. What it looks like, and where it belongs.

One single vector holds both. With that, the transformer can finally tell a top left patch from a bottom right one. The order is baked in, and the picture keeps its shape. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the CLS token? Let's try to understand. In a simple way. By now, the image has become a row of tokens.

One vector for each little patch, all lined up in a sequence. But to label the whole image, the model needs one single vector that stands for all of it. And no single patch is the whole picture. So we add one extra token to the front of the sequence.

It is called the class token, or CLS token for short. This token does not come from any patch. It carries no picture content at all. It is just a learnable vector, trained along with the model.

Then it rides through the transformer, right beside all the patch tokens. It gets treated in exactly the same way as they do. As the layers run, this token looks at every patch and pulls in what matters. Slowly, it fills up with a summary of the whole image.

At the very end, we read out just this one token. Its final vector is the model's summary of the entire picture, ready for the label. Why not just average all the patches instead? You could, but the CLS token learns how to blend them.

That is a smarter summary than a plain average. So, the CLS token is one extra learnable slot. It gathers from every patch, and its final vector speaks for the whole image. Quick check now.

One question is coming up. Let's see if it clicked.

How do image patches look at each other? Let's try to understand. In a simple way. By now, the image is a row of patch tokens.

Each one is a small vector that describes just its own little square, and nothing else around it. But a patch on its own can be unclear. A patch of brown could be fur, or tree bark, or plain dirt. Only the patches around it settle what it really is.

So the transformer lets every patch look at every other patch. This step is called self attention. Each patch gathers clues from across the whole picture. Not every patch matters equally.

Each one gives more weight to the patches that are relevant to it, and mostly ignores the rest. Then the patch updates itself. Its new vector is a weighted blend of all the patches, leaning hard on the few that truly matter. And this happens for every patch at the same time.

All of them look, weigh, and update together, in one parallel step. There is no tiny window here. A patch at the top left can look straight at one in the bottom right, all in a single step. After this, no patch stands alone.

Each one now carries its own look plus the context around it. The patches finally understand each other. So, self attention lets every patch look at every other one, weigh what matters, and blend it in. Each patch becomes aware of the whole picture.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a patch's query, key, and value? Let's try to understand. In a simple way. You already know attention lets each patch look at the others.

But to do that, every patch first turns itself into three separate vectors. A query, a key, and a value. The query is a patch asking a question. It means, here is what I am looking for in the rest of the image.

The key is a patch's label. It advertises what that patch contains, so others can judge whether it is worth paying attention to. The value is the real information a patch will hand over. Its actual content, ready to be shared if another patch decides to pull it in.

Where do the three come from? Each patch vector is multiplied by three learned weight matrices. One builds queries, one builds keys, one builds values. The same three for every patch.

Now the matching. A patch takes its query and compares it against every other patch's key. A close match scores high. A weak match scores low.

Those scores decide the mix. The patch pulls a lot of value from the keys that matched well, and almost nothing from the rest. Picture a library. Your query is the search you type.

Each book's key is the title on its spine. The value is the words inside, what you actually walk away with. So one patch plays three roles. The query asks, the key answers, the value delivers.

Match queries to keys, and the right values flow in. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a transformer block? Let's try to understand. In a simple way. By now, our image is a row of patch tokens, and attention lets them look at each other.

But that is only one step. A block packages it all together. Inside one block there are two main parts. First, an attention sublayer where patches share information.

Then a small network applied to each patch. The first part is the self attention we already met. Every patch looks around, weighs what matters, and blends in context from the rest. The second part is a small feed forward network.

It runs on each patch by itself, with no mixing, letting that patch digest what it just gathered. Around each part sits a skip connection. The output is added back onto the input, never replacing it. So the original patch is kept, and learning stays stable.

And just before each part, a step called layer normalization gently rescales the numbers. It keeps the values in a healthy range, so the block trains smoothly. Put it in order. Normalize, then attention, then add back.

Normalize again, then the feed forward network, then add back. That whole path is one block. A block takes a set of tokens in, and hands back the same number of tokens, each the same size. Refined, but shaped just as they came in.

So one block is attention to mix the patches, then a feed forward network to refine each one, each wrapped with normalize first and a skip connection. Quick check now. One question is coming up. Let's see if it clicked.

How does stacking layers work? Let's try to understand. In a simple way. Remember one block.

It takes our patch tokens in, and hands the same tokens back, refined. Same count, same size, only richer inside. Because the shape never changes, the output of one block fits perfectly into the next. So we stack many blocks in a row, one after another.

The tokens leaving block one become the input to block two, then block three, and onward. Every block re-reads them and refines them a little more. This build up creates a hierarchy. Early blocks pick up simple things, edges and rough textures.

Deeper blocks fold those hints into richer parts, an eye, a wheel, a face. Why not stop at one block? Because one block can only refine once. Stacking lets each layer build on the last, so simple patterns slowly grow into whole concepts.

One key point. The stacked blocks are not copies. Each one has its own learned weights, so every layer can specialize in a different job. How many do we stack?

A common vision transformer uses twelve blocks in a row. Bigger models use twenty four or more. Depth is a design choice. So the deeper a token travels, the more abstract it becomes, from raw pixels toward real meaning.

That is the payoff of depth. More layers, more understanding. So stacking works because each block keeps the shape. Many blocks, each with its own weights, turn simple patterns into deep, meaningful features.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the classification head? Let's try to understand. In a simple way. After every transformer block has run, one token holds a summary of the whole image.

The CLS token. Its final vector is what we care about. So we ignore all the patch tokens now. We read out just the CLS token, a single vector of numbers.

That vector is our starting point. But a list of numbers is not an answer. We want a word. A class, like cat, or dog, or car.

Something we can read. So on top, we bolt one small layer. Just a single linear layer. It is called the classification head, and it does one job.

It turns that one vector into a row of scores. One score for every possible class. A thousand classes means a thousand scores. These raw scores are called logits.

The scores are messy, so softmax squeezes them into probabilities. Every class gets a value between zero and one, and together they add up to one. Now the answer is easy. We pick the class with the highest probability.

That winning class is the label the model gives the image. Why feed the head the CLS token and not a patch? Because the CLS token spent every layer gathering the whole image, shaped exactly for this final decision. So the final CLS vector goes into one small head.

Scores, then softmax, then pick the top one. That is how a token becomes a label. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is the difference between local views and global context? Let's try to understand. In a simple way. In a convolutional network, each filter looks at only a tiny window of the image.

A small patch of nearby pixels. That narrow view is called its receptive field. To see more of the image, you stack layers. With each layer the receptive field grows a little.

So distant corners only meet deep in the stack. A vision transformer works differently. Through self attention, any patch can look at any other patch, right from the very first layer. So distance costs nothing.

A patch in the top left corner reaches one in the bottom right in a single step, not after climbing through many layers. Put them side by side. The convolutional network builds up to a global view slowly, layer by layer. The transformer has that global view from the start.

But the local view is not a weakness. It bakes in a useful assumption, that nearby pixels belong together. The transformer starts with no such hint and must learn every relationship itself. Global context also has a price.

Every patch compares itself to every other patch. So the work grows with the square of the number of patches. So neither one simply wins. Local views are efficient and structured.

Global context is flexible and far reaching. The best models often borrow from both. So, convolution sees locally and reaches far only with depth. Attention sees the whole image at once.

Local structure versus global reach. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a Vision Transformer's hunger for data? Let's try to understand. In a simple way. A convolutional network comes with rules baked into its wiring.

Nearby pixels belong together. A shape means the same thing wherever it appears. Assumptions baked into the architecture. A Vision Transformer has almost none of those rules.

From the first layer, attention lets any patch look at any other. It never assumes neighbors matter more. So about images, it starts blank. It has to learn the basics from examples.

That nearby patches relate. That a cat is a cat wherever it sits. Learning those rules from scratch takes data. Not a little.

A mountain of it. Every example is one more clue about how images actually behave. Give it a small dataset, say a million images, and a convolutional network beats it. When data is scarce, those baked-in rules are a real head start.

Now feed it a giant dataset. Hundreds of millions of images. The transformer catches up, then pulls ahead. With enough examples it beats the hand-built rules.

Why the flip? Fewer baked-in assumptions means more flexibility. But flexibility only pays off with enough data to pin it down. Too little, and that freedom just overfits.

So the recipe is simple. Pretrain the model on a massive dataset first. Then fine-tune on your smaller task. That is how Vision Transformers became practical.

So the deal is this. A Vision Transformer trades built-in image assumptions for raw flexibility. And that trade only wins with a mountain of data. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is big data's edge for transformers? Let's try to understand. In a simple way. Picture a graph.

A model's skill, against how much data it sees. Each model draws its own curve. A convolutional network starts strong. Its built-in rules are a head start.

But those rules are also a ceiling. Add more data, and the curve flattens. A transformer starts slower, with few built-in rules. But it has room to keep learning.

Feed it more, and its curve just keeps rising. Why the difference? A transformer learns its rules from data, instead of having them wired in. More examples, sharper rules, until they beat the hand-built ones.

With so little baked in, there is always more structure to find. So it does not saturate. Even at hundreds of millions of images, no ceiling yet. It also scales cheaply.

Every layer is the same, and attention runs in parallel. Easy to grow the model and train on enormous data. Two dials, really. More data needs a bigger model to use it.

A bigger model needs more data to fill it. Transformers turn both. The payoff is so steady it has a name. Scaling laws.

Add data and compute, and accuracy climbs a smooth, predictable line. The same hunger shows up in language, vision, and audio. A flexible learner, fed a mountain of data. That is why transformers took over.

So, a transformer keeps turning extra data into new rules. A CNN plateaus; the transformer does not. Give it big data, and it keeps getting better. Quick check now.

One question is coming up. Let's see if it clicked.