Deep learning's greatest hits

What exactly is a single neuron? Let's try to understand. In a simple way. A single neuron is a tiny decision maker.

It looks at a few clues, pieces of evidence, and turns them into one number. The question here: is this email spam? Each clue gets its own weight. A weight is just a number saying how strongly that clue counts.

A big weight means the neuron really trusts that clue. For every clue, the neuron multiplies the evidence by its weight. That gives each clue a vote. Money, times a weight of two, casts a vote of two.

Then it adds every vote into one number, the weighted sum. Two plus three minus one comes to four. That is the core of the neuron. Now the signs.

A positive weight is evidence in favor. The many links clue is positive, so seeing links pushes the number up. A negative weight is evidence against. A known sender has a negative weight, so it pulls the spam number down.

Weights can argue both ways. There is one more piece, the bias, a baseline lean the neuron holds before any evidence arrives. Start skeptical, say minus one, and the score settles at three. That final number is the neuron's verdict.

The higher it climbs, the more convinced it is. It does not decide yet. It simply passes the score onward. So a neuron weighs evidence.

Multiply each clue by its weight, add the votes, add a baseline. One number comes out, the neuron's take on the question. Quick check now. One question is coming up.

Let's see if it clicked.

How does gradient descent work? Let's try to understand. In a simple way. A fresh network is just a big box of adjustable knobs.

At the start they are set at random, so its very first guesses are basically nonsense. So you show it one example and let it guess. Say the true answer is ten, and the network blurts out three. Clearly off.

Measure how wrong that was. The gap between the guess and the truth becomes a single number, the error. The bigger the gap, the bigger the error. Here is the trick.

Learning from mistakes means using that error to fix the knobs. Every knob gets asked, did you push the guess too high or too low? For each knob, we work out which way to turn it to shrink the error. That set of directions has a name.

It is the gradient. Picture the error as a hilly landscape. Your knobs put you at one spot. The gradient points straight uphill, so to lower the error you head the opposite way, downhill.

Take a small step downhill, not a wild leap. Leap too far and you overshoot the bottom. That careful step size is called the learning rate. Then repeat.

Guess, measure the error, step downhill, again and again, over thousands of examples. Each pass the error shrinks a little, and the guesses get better. So gradient descent is learning from mistakes. Guess, measure the error, and nudge every knob a small step downhill.

Repeat, and the network teaches itself. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a nonlinear activation function? Let's try to understand. In a simple way. Start with what one layer really does.

It multiplies its inputs by weights, adds them up, and passes the total along. That is just a straight line transform. Now stack a second layer on top, doing the same thing. Here is the catch.

A straight line feeding another straight line is still just one straight line. So without anything in between, a hundred stacked layers collapse into a single line. All that depth, completely wasted. The network can only draw straight.

The fix is to slip a nonlinear function between the layers. It takes each neuron's number and gently bends it. This is the activation function. The most common one is relu.

Its rule is tiny. Negative numbers become zero, and positive numbers pass straight through. One simple kink in the line. One kink does not sound like much.

But with many neurons, each adding its own bend, the network can piece together almost any curve it needs. Now it can separate tangled data that no straight line could ever split. Curved boundaries, loops, and islands all become reachable. That is why these functions are the key.

Without the bends, extra depth buys you nothing. With them, a deep network can model almost anything. So, nonlinear activations are the bends between the layers. They stop a network collapsing into one line, and unlock its real power.

Quick check now. One question is coming up. Let's see if it clicked.

How does a sliding filter work? Let's try to understand. In a simple way. To a computer, a picture is just a grid of numbers.

Each square is a pixel, a brightness value. No lines or shapes, only numbers. You could wire every pixel to its own neuron. But a small photo already has hundreds of thousands of pixels.

Far too many connections to learn. So instead we use a tiny filter. It is a small grid of weights, three by three. It only looks at a small patch at a time.

That filter slides across the image, one step at a time. Left to right, top to bottom. It visits every patch. At each stop, it multiplies each pixel by the weight on top of it, then adds them up.

Nine numbers become one. That single number says how strongly the patch matches the filter's pattern. One filter lights up on edges. Another on a curve.

Here is the clever part. The same filter is reused at every spot. So a pattern is caught wherever it appears, with far fewer weights to learn. All those numbers form a fresh grid, called a feature map.

Bright spots mark where the pattern was found. A map of one feature. A layer does not stop at one filter. It runs many at once.

Each hunts for its own pattern, so the picture becomes a stack of feature maps. So a filter is a small pattern detector that slides everywhere. Multiply, add, and it charts where each pattern lives. That is how computers see.

Quick check now. One question is coming up. Let's see if it clicked.

How does pooling work? Let's try to understand. In a simple way. After the sliding filters run, you are left with a big grid of numbers.

A feature map. It works, but it is a lot to carry forward. Pooling shrinks that grid. It slides a small window across the map, usually just two by two, and looks at four numbers at a time.

For each little window, it throws away all but one number. Max pooling keeps the biggest value it finds, and forgets the other three. The window steps over by two each time, so the windows never overlap. Width and height are halved.

The grid drops to a quarter of its size. There is a softer flavor too. Average pooling takes the mean of the four numbers instead of the biggest. Same shrinking, a gentler summary.

So why keep the maximum? A big value means the filter found its feature right there, strongly. Keeping the max keeps the strongest evidence and drops the rest. Here is the quiet bonus.

If that feature slides one step sideways, it still lands in the window, so the max barely changes. Small position wobbles stop mattering. And pooling has nothing to learn. No weights, no training.

It is a fixed rule that runs the same way every single time. So, pooling shrinks the feature map, keeps the strongest signals, and lets small shifts slide. Fewer numbers, and a tougher network. Quick check now.

One question is coming up. Let's see if it clicked.

How does a network learn to remember sequences? Let's try to understand. In a simple way. Think about a sentence.

The words arrive one after another, and their order carries the meaning. A plain network sees everything at once, with no sense of what came before. So feed it the words one at a time instead. But a plain network has a problem.

After each word, it forgets. The next word arrives to a blank slate. The fix is to give the network a small memory. A little bundle of numbers called the hidden state.

It carries across from one step to the next. At every step the network combines two things. The new input word, and the memory from the step before. Out comes a fresh, updated memory.

That updated memory then loops back in for the next word. This feedback loop is the whole trick. It is why we call this a recurrent network. And it is the same tiny network running at every step.

One set of weights, reused again and again, no matter how long the sequence grows. Stretch it out across time and you see a chain. One copy per word, with the memory handed along the links like a note passed down a line. So the hidden state is really a running summary.

A compressed picture of everything the network has read so far, ready for the next word. That is how a network remembers a sequence. A memory carried forward, updated at each step, by the same repeating loop. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the problem of a fading memory? Let's try to understand. In a simple way. Remember how a recurrent network reads.

One word at a time, updating a small memory that it carries forward. Unrolled across time it is a chain. Each word hands its memory to the next, like a note passed down a long line of people. But sometimes the clue you need sits far back.

I grew up in France, so I speak fluent, blank. The answer, French, depends on a word many steps earlier. The trouble is that memory is one small fixed bundle. At every step the new word is blended in, and older details get written over.

Each step also multiplies what is already stored. Multiply by a number below one, again and again, and the early signal shrinks quickly toward zero. The same shrinking hurts learning. To link two far apart words, the training signal travels back through every step and fades at each one.

This is the vanishing gradient. So by the time the network reaches the blank, the word France has almost faded away. The clue is gone, and the guess falls flat. Nearby words stay fine.

It is the long range links that break. The farther apart two words sit, the weaker the memory that survives. So a plain recurrent network holds the recent past well, but its grip on the distant past fades. That is the problem of a fading memory.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a memory gate? Let's try to understand. In a simple way. A plain recurrent network keeps one running memory.

But every step rewrites it, so older details wash away. It struggles to hold onto what matters. So we add a second track. A memory line that runs straight through, almost untouched.

Picture a conveyor belt carrying information forward. Along the belt we place gates. A gate is a tiny learned valve. For each memory value it outputs a number from zero to one.

Zero blocks, one lets it pass. The first gate decides what to erase. It picks which values to keep and which to wipe. This is the forget gate.

The second gate decides what to add. It chooses which pieces of the new input get written onto the belt. This is the input gate. The third gate decides what to reveal.

It passes out only the part of memory that matters right now. This is the output gate. Here is the trick. When the forget gate stays near one, a value rides the belt unchanged for many steps.

A detail from the start can reach the end. And every gate is learned. From the data, the network works out when to open it and when to close it. A belt guarded by these three gates has a name.

Long short term memory. The gates are what let it hold on. So, gates are learned valves on a memory belt. Forget clears, input writes, output reveals.

That is how a network holds on. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a word embedding? Let's try to understand. In a simple way. A model only speaks numbers.

A word has an id, its slot in the vocabulary. But that id is just a name tag. Slot five is no closer to slot six. So instead we hand each word a short list of numbers.

That list is its vector, the word's own coordinates. Coordinates place a point in space. Two numbers pin a spot on a map. Real models use hundreds, so each word sits in a vast space.

Here is the payoff. Words with similar meaning end up near each other. Cat lands close to dog, king close to queen. Distance now means something.

But who picks these numbers? Nobody. At the start every word is dropped at a random spot. The coordinates are pure noise.

Then training takes over. The coordinates are just more weights, so the same gradient descent nudges them, little by little, until related words drift together. Once trained, directions carry meaning too. Step from man toward woman, take that same step from king, and you land near queen.

Where does this live? One big table, one row per word, and the row is its vector. To embed a word, read its row. This lookup table is the model's first layer.

Token ids come in, coordinates come out, and every later layer works on those coordinates. So a word becomes a vector of trainable coordinates. Random at first, then shaped by training, until nearness and direction both carry meaning. Quick check now.

One question is coming up. Let's see if it clicked.

How does compressing information through a bottleneck work? Let's try to understand. In a simple way. Picture a photo going into a network.

It is really a huge list of numbers, one for every pixel. Most of that detail we do not really need. Now force all of it through a tiny middle layer. Only a handful of numbers wide.

That narrow squeeze is the bottleneck. The first half of the network is the encoder. Its job is to pack the big input down into those few numbers. The second half is the decoder.

From that tiny code alone, it tries to rebuild the original photo. We train the whole thing on one simple goal. The output should match the input. The gap between them is the error we shrink.

Here is the clever part. The code is too small to hold everything. So the network cannot simply copy. It has to keep only what matters.

It learns to throw away noise and repetition, and store the essence. The shapes and layout, the few things that truly define the picture. That small middle layer is now a compressed summary. A short code that captures the input in far fewer numbers.

This is why bottlenecks show up everywhere. They compress data, clean up noise, and flag the odd input that will not rebuild well. So a bottleneck squeezes information down, then back. Forced to stay small, the network learns to keep only what really counts.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the trouble with one word at a time? Let's try to understand. In a simple way. A recurrent network reads a sentence one word at a time, the way you do.

Each word updates a running memory before the next arrives. But here is the catch. Step three cannot begin until step two finishes. Step two waits on step one.

Every step is chained to the one before it. So the words are handled strictly in order. You cannot jump ahead. The network is stuck walking the sentence in single file.

Here is why that hurts. GPUs can do thousands of things at once. But a chain cannot be split apart, so most of that power sits idle. And it gets worse as sentences grow.

Twice as long means twice as many steps in a row. The waiting climbs with every word. There is a second cost. For the first word to reach the fiftieth, its signal must pass through every step in between.

Distant words sit far apart. What we really want is to look at every word at the same time. Read the whole sentence in one shot, not one step after another. That is exactly what came next.

A design that drops the chain and handles every word in parallel. It needs a new way for words to share information. So, the trouble with one word at a time is the chain: strictly in order, hard to parallelize, slow on long text. Break it, and the whole sentence flies.

Quick check now. One question is coming up. Let's see if it clicked.

How does attention work? Let's try to understand. In a simple way. Read this sentence.

The animal didn't cross the road because it was too tired. What does it refer to? To know, you must look at the other words. That is the whole idea of attention.

Every word gets to look at every other word, and decide how much each one matters to it. How does a word decide? It sends out a query. Every other word answers with a key.

Match query to key, and you get a score. So the word it scores every word in the sentence. Animal scores high, it is what we are talking about. Road and tired score low.

Next, squash those scores into weights that add up to one. Think of it as splitting one whole point of attention across all the words. Each word also carries a value, the actual information it offers. The weights decide how much of each value to take.

Now blend. Take a big slice of animal, a sliver of the rest, and add them up. That weighted sum becomes a fresh vector for it. And this is not just for it.

Every word builds its own blend at the same time, each one reweighing the whole sentence for itself. The weights are computed fresh for every sentence. So the same word can attend to completely different neighbours, depending on what surrounds it. So attention lets each word weigh every other word, turn those weights into a blend, and walk away carrying the context it needs.

Quick check now. One question is coming up. Let's see if it clicked.

How does positional encoding work? Let's try to understand. In a simple way. Attention looks at every word in a sentence at the same time.

All at once, in parallel. Not one after another. But here is the catch. On its own, attention treats those words as a loose bag.

It cannot tell which came first, or last. Take dog bites man, and man bites dog. Same three words, opposite meaning. Yet to plain attention, both look the same.

So we give each slot in the line its own vector. A position vector. We add it onto the word's embedding, before attention looks. Every position gets its own unique pattern.

Slot one, slot two, slot three, each a different fingerprint. So a word carries where it sits. The original recipe builds those patterns from waves. Sine and cosine curves, some fast, some slow.

At each slot they give a combination that never repeats. Notice we add the position in, we do not staple it on the end. The vector keeps its length. Meaning and position now live blended together.

Newer models often skip the fixed waves. Instead they learn a position vector for each slot, the way they learn word meanings. Training decides what each spot says. Now the same word in slot one versus slot three carries different numbers.

So dog bites man and man bites dog look different at last. So, positional encoding hands every slot its own vector, and adds it to the word. Now attention reads all words at once, and still knows their order. Quick check now.

One question is coming up. Let's see if it clicked.

How does multi-head attention work? Let's try to understand. In a simple way. You already know attention.

Each word looks at the others and blends in what matters. But one pass picks only one way to relate the words. The trouble is, a sentence holds many links at once. Which noun a pronoun points to.

Which verb goes with which subject. One view cannot catch them all. So the transformer does not run attention once. It runs several copies side by side.

Each copy is called a head. A model might use eight of them. Here is the key part. Every head gets its own query, key, and value weights.

So each head builds its own questions, and scores the sentence its own way. Because those weights differ, each head learns a different angle. One head tracks what a pronoun refers to. Another follows the verb.

Another watches the next word. All the heads read the same sentence at the same moment. None of them waits for the others. Each hands back its own small vector for every word.

Then, for each word, the heads' outputs are lined up side by side, and passed through one more weight matrix. That mixes the angles into a single vector. Now every word carries many relationships at once, not just one. That richer picture is exactly why many heads beat a single one.

So multi-head attention runs attention several times in parallel, each head on its own angle, then blends them back together. Quick check now. One question is coming up. Let's see if it clicked.

How does the encoder to decoder handoff work? Let's try to understand. In a simple way. A translation model comes in two halves.

An encoder that reads the input, and a decoder that writes the answer, one word at a time. The encoder reads the whole input at once, and turns it into a set of rich vectors. One per word. Think of it as a memory of the sentence.

The decoder writes the output word by word. But to say the right thing, it must glance back at the input. So it talks to the encoder. That conversation is called cross attention.

The decoder asks a question. The encoder's memory answers. It is attention, but across the two stacks. Here is who says what.

The query comes from the word the decoder is writing now. The keys and values come from the encoder's memory. Different sources, one match. So the decoder scores every input word, softmax turns those into weights, and it blends the encoder's values.

The next output word leans on the right part of the input. And this repeats at every step. Writing one word may lean on one input word. The next may lean on another.

Fresh focus each time. One more thing. The encoder runs only once. The decoder reads that same memory again and again, at every step.

Encode once, read many times. So, the encoder folds the input into a memory, the decoder writes word by word, and cross attention is the phone line carrying meaning between them. Quick check now. One question is coming up.

Let's see if it clicked.