How machines see

How does a computer turn a picture into numbers? Let's try to understand. In a simple way. Look at any photo on a screen.

To you it is a face, or a sky. But a computer has no eyes. Inside, it only stores numbers. So zoom in, really close.

The smooth picture breaks into a grid of tiny squares. Each little square is called a pixel. In a black and white image, each pixel holds one number, its brightness. Zero is fully black, two fifty five is fully white, and the greys sit in between.

Line those numbers up, and the whole image becomes a grid of values. No shapes, no lines. Just rows and columns of brightness. But most pictures are in color.

For color, one number is not enough. Each pixel now needs three, how much red, how much green, and how much blue. Mixing those three amounts can make any color you see. So a color image is really three grids stacked together.

We call them the red, green, and blue channels. Put it together, and a photo is just a big block of numbers. Width, times height, times three. That block is everything the computer ever gets.

And more pixels means more numbers, and finer detail. That count is the resolution. A tiny icon is a few hundred numbers; a full photo is millions. So, to a computer, an image is not a picture at all.

It is a grid of numbers. Brightness for grey, and three values for color. Quick check now. One question is coming up.

Let's see if it clicked.

How does a filter find edges in a picture? Let's try to understand. In a simple way. First, what is an edge?

Look at the border of any object. One side is bright, the other is dark. That sudden change in brightness is an edge. To a computer the image is just brightness numbers.

Inside a smooth wall, neighboring pixels are nearly the same. At an edge, two neighbors differ a lot. So we build a tiny grid of numbers called a filter. A simple one holds a minus one on the left and a plus one on the right.

Here is the trick. The filter subtracts the left pixel from the right one. It measures how much the brightness changed between them. On a flat patch both sides are equal, so the difference is nearly zero.

At an edge, one side is bright and one is dark, so the difference is large. Now slide that filter over every spot in the image. Each spot gives one number. Together they form a new grid, dark on flat areas and bright right on the edges.

Direction matters too. This left to right filter reacts to vertical edges. Turn the same idea on its side, and it catches horizontal edges instead. The sign of the number even tells you which way the edge goes.

Dark to light comes out positive. Light to dark comes out negative. So an edge is just a jump in brightness, and a filter finds it by subtracting neighbors. Flat stays quiet, a jump rings loud.

Quick check now. One question is coming up. Let's see if it clicked.

How does a machine stack filters to see shapes? Let's try to understand. In a simple way. Start with one filter.

A single filter finds one simple thing, like an edge. But an edge is not a shape. So what is a shape, really? An eye is curves and corners.

A wheel is a circle inside a circle. Every shape is smaller parts, arranged together. One filter is never enough. The trick is to stack filters in layers.

Each layer's output becomes the next layer's input. The first layer looks at the raw pixels. Its filters find the simplest things: short edges and little lines. Each filter slides across the image and leaves a new grid.

That grid is a feature map. It lights up wherever its pattern was found. Now the clever part. The second layer does not look at pixels.

It looks at the feature maps from the first. Combining edges, it finds corners and curves. Stack another layer, and it combines those into parts. An eye.

A wheel. A petal. Real pieces of an object. The deepest layers put the parts together into whole things.

A face. A car. Parts became objects. That ladder is a feature hierarchy.

And why does depth help? A filter deep in the stack covers a large patch of the image, so it reacts to big shapes. Nobody draws these filters by hand. The network learns them.

So, to see a shape, a machine stacks filters. Each layer builds on the one before. Simple edges turn into full objects. Quick check now.

One question is coming up. Let's see if it clicked.

How does a network decide if it sees a cat? Let's try to understand. In a simple way. By now, earlier layers have boiled the image down to a short list of clues.

Pointy ears here, whisker lines there, a patch of fur. These clues feed the final decision. The network is never asked a plain yes or no. It carries a fixed list of labels it was trained on.

Cat, dog, car, and so on. Its job is to choose one. So the last layer hands every label a score. Just one number, for how well the clues match that label.

Cat might earn a big number, car a tiny one. Where does a score come from? It is a weighted vote. Clues that shout cat, like whiskers and ears, push the cat score up.

Clues that fit a car stay silent. Raw scores are hard to compare, so they get squeezed into percentages that add up to one hundred. Now cat sits at eighty, dog at fifteen, the rest tiny. That spread is the network's confidence.

The decision itself is easy. Whichever label holds the highest percentage wins. Cat at eighty beats every other label, so the network finally answers, cat. But it will crown a winner even when it is unsure.

If cat and dog both hover near forty, no score is strong. A careful system reads that as, not confident enough. So deciding cat is really scoring. Give each label a number, turn the numbers into confidence, and pick the highest one.

Quick check now. One question is coming up. Let's see if it clicked.

What does a machine see when it gets confused? Let's try to understand. In a simple way. A vision model always sorts a picture into labels it already knows.

For every label, it gives a confidence score, a percent. And they always add up to one hundred. When the picture is clear, one score towers over the rest. Cat, ninety five percent.

The model is sure, and it is right. Confusion looks different. Two scores come out almost equal. Cat, forty.

Fox, thirty eight. Neither one clearly wins. Why does that happen? The clues it noticed, pointy ears, fur, a small nose, fit more than one animal.

So the confidence gets shared. Here is the catch. The model has no I do not know button. It must spread one hundred percent across the labels it was taught.

So show it something it never saw, like a blurry pastry, and it still picks a label. It cannot say none of these. And a high score is not proof. A model can be ninety nine percent sure and still be wrong.

Change a few pixels, and its answer flips. So people watch the gap between the top two scores. A wide gap means sure. A thin gap means the machine is confused.

So a confused machine is not blank. It is torn between labels, spreading its confidence, sometimes sure and still wrong. Quick check now. One question is coming up.

Let's see if it clicked.

How does a machine label every single pixel? Let's try to understand. In a simple way. Normal classification gives the whole picture just one label.

Cat. But that throws away where things are. It cannot tell the cat from the couch behind it. Segmentation asks for more.

Not one label for the image, but a label for every single pixel. Cat here, couch there, floor below. So the network handles each pixel on its own. For that one spot, it produces a score for every class.

Cat, couch, floor, and the rest. Then it does what any classifier does. It keeps the highest score. That winning class becomes the label for that one pixel.

Repeat that for every pixel, and you get a label map. Same width and height as the photo, but each cell now holds a class, not a color. Paint every class its own color, and the map becomes a mask. All the cat pixels glow one shade, all the floor pixels another.

For the labels to line up, the network cannot squash the image into one blob. It keeps the layout, shrinking it, then growing it back to full size. One catch. Every cat pixel simply says cat.

If two cats touch, this map paints them as a single blob. Telling them apart is another job. So labeling every pixel is classification, run everywhere at once. Score each pixel, pick its winner, and color the whole picture in.

Quick check now. One question is coming up. Let's see if it clicked.

How does a machine separate one car from another? Let's try to understand. In a simple way. Start with a picture where every car pixel just says car.

When two cars touch, they melt into one giant car shaped blob. But we usually want to count them and follow each one. Is that one long car, or three parked bumper to bumper? A blob cannot say.

Instance segmentation fixes this. Instead of one label for the whole class, it gives each separate object its own mask. Car one here, car two there. How?

First it finds each object as its own thing, one region per car. Then it paints a mask inside each region, tracing that car alone. So every car gets its own identity tag. Same class, car, but a different number.

Car one, car two, car three, each its own color. Because each mask is built on its own, objects can overlap. A car parked behind another still keeps a separate mask, even where they touch. Compare the jobs.

Labeling pixels alone paints one solid blob. Separating instances splits it into three cars you can actually count. This is why a self driving car needs it. It must track this car and that car as separate objects, not one blurry mass of metal.

So separating cars means giving each object its own mask and its own tag. Find each one, outline it, and now the machine can count and follow them. Quick check now. One question is coming up.

Let's see if it clicked.

How does a network reconstruct the full image details? Let's try to understand. In a simple way. To understand a picture, a network first shrinks it.

Layer by layer, it squashes the image into a small, coarse map that says what is there. But shrinking throws away detail. That tiny map knows there is a cat, but the exact edges, the fine outline, are gone. Yet the output we want is full size.

A crisp mask matching the original image, pixel for pixel. So the second half of the network grows the map back up, step by step, toward full resolution. This half is the decoder. But stretching a tiny map alone stays blurry.

Enlarging coarse cells just smears them wider. The what is back, but the sharp where is missing. The fix is a shortcut. The early layers still hold the sharp, high resolution detail.

The network copies those features straight across. These bridges are skip connections. Now the decoder blends two things. The coarse map that knows what, and the copied detail that knows where.

Together they rebuild sharp, clean edges. Drawn out, the path goes down then back up, with bridges across the middle. That shape gives this design its name, the U net. This is why segmentation masks hug the true boundary, and why a scan can be traced tumor tight.

Detail lost going down is restored going up. So reconstructing detail means two halves. Shrink to learn what, then grow back up while copying in the fine detail to nail exactly where. Quick check now.

One question is coming up. Let's see if it clicked.

How does a machine draw a box around a person? Let's try to understand. In a simple way. A classifier can tell you a person is in the picture.

But it does not say where. Somewhere in this frame is not very useful. So detection adds a location. It draws a box, a rectangle pulled tight around the person, so you can point right at them.

Here is the trick. That box is really just four numbers. Where its top left corner sits, and how wide and how tall it is. Each box carries two more things.

A label, saying person, and a confidence score, saying how sure the network is about it. How does it get those numbers? It does not paint any pixels. It predicts the four coordinates directly.

Guessing numbers like this is called regression. It learns from examples. People drew the true box by hand. The network compares its guess to that box, and nudges until the two line up.

A box is coarser than a pixel mask. It does not trace the exact outline. It just brackets the person inside a simple rectangle. This is everywhere.

A phone finds faces to focus the camera. A car spots a pedestrian. Each one is a quick rectangle with a name. So drawing a box means four numbers and a label.

Predict the coordinates, check them against the human box, and the machine can finally say, there. Quick check now. One question is coming up. Let's see if it clicked.

Why does a network guess many boxes for one object? Let's try to understand. In a simple way. To find objects, a detector does not check one spot.

It scans the whole image, testing boxes of many sizes at many positions. Thousands of possible boxes. Now picture one dog. Every box that lands near it, covering most of the dog, fires the same answer.

Object here. So a single dog lights up many overlapping boxes. The raw output is a mess. Ten or twenty boxes stacked on the same animal.

But we wanted one clean box around the dog, not a messy heap. The cleanup uses a number every box already carries. Its confidence score. How sure the network is that a real object sits inside that box.

Step one. Pick the winner. The box with the highest confidence, since that one usually hugs the dog the best. Keep it.

Step two. Compare every other box to the winner. Measure how much they overlap. A lot of overlap means they are framing the very same dog.

So drop them. Delete every box that overlaps the winner too much. This is called non maximum suppression. Then move to the next object.

Pick the top box that remains, suppress its overlappers, and repeat. When it ends, each object keeps just one box. So many boxes appear because many nearby spots vote for the same object. Keep the highest score, suppress the boxes that overlap it, and you get one object, one box.

Quick check now. One question is coming up. Let's see if it clicked.

How does a network find all objects in one glance? Let's try to understand. In a simple way. The old way was slow.

Slide a little window across the picture, and run the whole classifier on each spot. Thousands of little checks, one after another. The one glance idea flips that. Look at the entire image just once.

A single pass through the network, and every object comes out together. The trick is a grid. The network lays a grid of cells over the whole picture, splitting it into a neat set of squares. Now every cell gets one small job.

It asks, is the center of an object sitting inside me? If yes, it predicts a box, a label, and a confidence. And here is the key. Every cell answers at the same moment, in that single pass.

Hundreds of little predictions, all produced together, not one at a time. Which cell owns a given object? The one the object's center falls into. That cell is responsible for drawing its box.

Most cells see nothing, so their confidence is near zero. Throw those away. Merge any leftover boxes that overlap. What remains is one clean box per object.

Because it is just one pass, it is fast enough for live video. This family is called You Only Look Once. It runs on phones and self driving cars. So one glance means one pass over a grid.

Every cell predicts at once, weak guesses are dropped, and all the objects appear together. Quick check now. One question is coming up. Let's see if it clicked.

How does turning an image into a sentence work? Let's try to understand. In a simple way. The goal is bigger than a single label.

We do not want just the word dog. We want a whole sentence, a dog catching a frisbee. To do this, the model has two halves. One is an eye that reads the picture.

The other is a writer that turns what it saw into words. First, the eye. A vision network scans the image and squeezes it into a set of number vectors. Each captures a piece of what is where.

Now the writer takes over. It is a language model, like the one behind chat. It reads those image vectors and predicts the very first word. Then it writes one word at a time.

Each pick feeds back in, and it guesses the next. This repeats until the sentence is done. How does it stay on topic? Attention.

For each word, the writer looks at the matching part of the picture. To say frisbee, it focuses on the frisbee. Where does this skill come from? Millions of pictures, each paired with a human caption.

It learns to line up regions of an image with the right words. And it is not copying. It builds a fresh sentence each time, so it can describe a scene it never saw in that way. So an image becomes a sentence in two steps.

An eye turns the picture into vectors, and a writer spells out the caption, one word at a time. Quick check now. One question is coming up. Let's see if it clicked.

How do images and words learn a shared language? Let's try to understand. In a simple way. A picture is a grid of pixels.

A sentence is a string of letters. To a computer these are two different worlds, hard to compare. So we build two encoders. One reads the image and boils it down to a vector.

The other turns the words into a vector. Here is the trick. Both vectors are sent into the same space. A picture and a sentence become the same kind of thing, a list of numbers on one shared map.

Training uses matched pairs. A photo of a dog, with the caption, a dog. We pull those two vectors together, until they point the same direction. At the same time, wrong pairings are pushed apart.

The dog photo and the caption, a car, should land far away. This push and pull is called contrastive learning. How close counts as together? It is just an angle.

Vectors pointing the same way score high. Pointing apart scores low. That score is their similarity. A whole batch trains at once.

Line up many images against many captions in a grid. The true matches sit on the diagonal, and only those score high. Once the space is aligned, you can invent any new caption. Score the image against each one, and the closest caption wins.

No extra training required. So, two encoders feed one shared space. Matches are pulled together, mismatches pushed apart. Now a picture and a sentence finally speak the same language.

Quick check now. One question is coming up. Let's see if it clicked.

How does answering a question about an image work? Let's try to understand. In a simple way. Here is the task.

You give the model a picture, and a question in words. What color is the umbrella? It should answer. Red.

But notice, two very different inputs. A picture is a grid of pixels. A question is a string of words. So each input gets its own encoder.

A vision encoder turns the image into feature vectors. A text encoder turns the question into word vectors. Here is the heart of it. Both sets of vectors pour into one shared transformer.

The image and question finally meet and mix. Inside, the question steers attention. It points the model at the region that matters. Ask about the umbrella, and it looks right at it.

Then a head reads that mix and scores a menu of possible answers. Red, blue, two, yes. The top score wins, and that is the answer. Same picture, new question, new answer.

How many people? Two. Is it raining? Yes.

The question decides what the model reports back. One catch. A lazy model could guess a common answer without looking. A good one checks the pixels, and answers what is really there.

Where does this come from? Millions of examples, each an image, a question, and its answer. It learns to match what is asked to what is shown. So, encode the image and the question, mix them in one transformer, let the question aim attention, and read off the top answer.

Quick check now. One question is coming up. Let's see if it clicked.

How does finding the dog you described work? Let's try to understand. In a simple way. Here is the task.

You type a phrase. The small dog on the left. And the model must draw a box around exactly that. Not pick a label from a list.

Match a whole description. Older detectors only knew a fixed list of labels. Person, car, dog. But your phrase is free text.

It could be almost anything you type. So first, your words get encoded. A text encoder turns the phrase into a vector. A list of numbers that captures its meaning.

Meanwhile the picture is broken into candidate regions. Little areas the model thinks might each hold an object. Now the heart of it. Each region becomes a vector too.

The model scores how well every region matches your phrase vector. Close vectors mean a strong match. The region with the highest score wins. The model draws the box right there.

That is your dog, found. And the exact words steer the choice. Ask instead for the dog on the right, and a different box wins. The description picks which region, not just what kind of thing.

Here is the payoff. Because it matches meaning, not a fixed label, you can ask for things it never had a class for. A red polka dot mug. People call this open vocabulary.

Where does this come from? Millions of pictures, each paired with a phrase and the box it points to. The model learns to line up words with the right region. So, encode your phrase, score it against every region, and box the one that matches best.

Quick check now. One question is coming up. Let's see if it clicked.