Neural networks from scratch

How does a single neuron work? Let's try to understand. In a simple way. A neuron is tiny.

It takes in a few numbers, called inputs. Picture the rooms, the size, and the age of a house, all fed in at once. Here is the key idea. Each input gets its own weight.

A weight is just a number that says how much that input should matter. The neuron multiplies every input by its weight. A large weight makes that input count for a lot. A weight near zero makes it barely count at all.

Then it adds all of those products together into one number. That running total is called the weighted sum. It is the heart of the neuron. The sign of a weight matters too.

Rooms has a positive weight, so adding rooms pushes the total up. Age has a negative weight, so it does the opposite and drags the total down. Weights can vote for an answer, or against it. This is how a neuron pays attention.

Inputs that matter get large weights. Inputs that do not get tiny ones. The weights decide what the neuron listens to. Line up the inputs, line up the weights, multiply pair by pair, then add.

Mathematicians call this a dot product. One clean number falls out. So a neuron weighs its inputs by multiplying each one by a weight and adding the results. That single weighted sum is the number it passes on.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a neuron's bias? Let's try to understand. In a simple way. First, a quick recap.

A neuron takes its inputs, multiplies each by a weight, and adds them all into a single number. But that number alone is not enough. Here is the missing piece. After the sum, the neuron adds one more number.

A constant, called the bias. It is added every time, no matter what the inputs are. Notice the bias has no input of its own. Weights are tied to inputs.

The bias is not. It is the neuron's own free number, standing on its own. So what does it do? It shifts the neuron's threshold.

A positive bias makes the neuron eager, firing on the smallest push. A negative bias makes it hard to please. Here is the tell. Turn every input to zero.

Without a bias, the sum is zero, so the neuron stays silent. With a bias, it can still speak up. Picture the neuron drawing a line that splits yes from no. The weights tilt that line.

The bias slides it across the space. Without a bias, it is stuck at the center. One more thing. The bias is not fixed by hand.

Like every weight, it is learned. Training slowly nudges it up or down, until the neuron fires at just the right moment. So, the bias is a constant the neuron adds on its own. It shifts when the neuron fires, free from any input.

It sets how easily the neuron speaks up. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is an activation function? Let's try to understand. In a simple way. A neuron adds up its inputs and lands on a single number.

Big, small, positive, or negative. But a raw number, on its own, is not a decision yet. So the neuron needs a rule. Should it stay quiet, or should it fire and pass a signal on?

That rule is the activation function. Think of it as a switch sitting at the neuron's exit. The weighted sum goes in, and the switch decides what actually comes out. The most common switch is called relu.

Its rule is tiny. If the number is negative, output a flat zero. If it is positive, let it pass through unchanged. So it is really a threshold at zero.

Below the line, the neuron is off and silent. Above the line, it is on, and bigger numbers pass through bigger. Not every switch is so sharp. A sigmoid is a soft dimmer.

It squashes any number into a smooth value between zero and one. Why bother at all? Without a switch, every neuron is just a straight line. And stacking straight lines only gives you another straight line.

The switch adds a bend. That bend is called nonlinearity, and it is the whole point. It lets the network curve, and trace shapes a straight line never could. So an activation function is the neuron's on off switch.

It decides who fires, and it is what makes a network more than a line. Quick check now. One question is coming up. Let's see if it clicked.

How does a layer of neurons work? Let's try to understand. In a simple way. A single neuron only answers one question.

It weighs its inputs and fires for a single pattern it has learned. Just one. But one pattern is rarely enough. To recognize a real thing, you need to spot many features at once.

Curves, edges, colors. So we place many neurons side by side. Each one is handed the very same inputs. And each looks for its own separate feature.

They all run at the same time, in parallel. They do not talk to each other. Each just quietly reports its own number. That whole row of neurons, reading one input together, is what we call a layer.

One input goes in, a list of numbers comes out. Here is the key. Every neuron in the layer has its own weights and its own bias. Same inputs, but a different question asked.

So a layer turns one input into a list of scores. If it holds sixteen neurons, you get sixteen numbers. One score per feature. And because they all share the same inputs, the whole layer is just one big multiply.

Fast, and easy for hardware to run. How many neurons you use is the layer's width. More neurons means more features it can hunt for at once. So a layer is many neurons in parallel, all reading the same input, each spotting its own pattern.

One in, many out. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a pattern detector? Let's try to understand. In a simple way. Quick recap.

A neuron takes each input, multiplies it by a weight, and adds them into one number. Those weights are the neuron's own dials. Here is the trick. Line those weights up in a grid, and they form a small picture.

A template. The pattern this neuron is hunting for. When an input matches that template, high where the weights are high, the sum grows large and the neuron fires. When it does not match, the sum stays small, and the neuron is quiet.

At the very start, the weights are random. The template is just noise, so the neuron lights up for no good reason. Then training begins. Every time the neuron guesses wrong, its weights get nudged a little.

Up where the pattern should be bright, down where it should be dark. Repeat this thousands of times, and the noise slowly sharpens. The template turns into a real pattern. Say a diagonal stroke, or a bright left edge.

Now the whole layer. It is not one neuron but many, side by side. Each one learns its own simple pattern. One catches a diagonal, another an edge, another a curve.

So a layer learns by shaping each neuron's weights into a template. Match the template, and the neuron fires. That is one layer, quietly becoming a bank of pattern detectors. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the handoff between layers? Let's try to understand. In a simple way. Start with one layer that has already done its work.

Every neuron inside it finishes and produces a single number. That number is the neuron's output. Now line those outputs up, side by side. Together they form a small list of numbers.

That list is the whole layer speaking at once. Here is the key move. That list does not just sit there. It is handed to the next layer, and it becomes that layer's inputs.

And it is handed over in full. Every neuron in the next layer receives every number in the list. Not a few of them, all of them. The next layer treats that list as fresh inputs.

It does its usual work, and produces its own new list of numbers. Now the pattern is clear. Each layer's output becomes the next layer's input. Layer after layer, the numbers keep getting passed along.

And it always moves in one direction, from the input side toward the output side. That single sweep has a name. The forward pass. One last thing to notice.

Nothing here is merely copied. Each handoff reshapes the numbers into a new form, a little closer to the answer. So layers talk by passing lists of numbers. One layer's outputs become the next one's inputs, flowing forward, step by step.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is depth in a neural network? Let's try to understand. In a simple way. Picture the first layer looking at an image.

On its own, it can spot only simple things. A short edge here, a patch of color there. Nothing more. Now stack a second layer on top.

Here is the trick. It does not look at the raw pixels. It looks at the edges the first layer already found. And it glues those edges together.

Two edges meeting become a corner. A few of them curving become an arc. Small parts, built out of simple lines. Add one more layer, and those parts join up again.

Corners and curves become a shape you would recognize. An eye, a wheel, a petal. See the ladder forming? Pixels, then edges, then parts, then whole objects.

Each rung sits higher, and means more, than the one below it. And every rung is built only from the rung just beneath it. A layer never sees the pixels again. It only reworks whatever the layer before it handed up.

So why not use one huge single layer? Because one layer jumps straight from pixels to the answer. It has no middle steps, so it only ever catches simple patterns. That is what depth buys you.

Each layer is a little more abstract than the last. Early layers stay generic, deep layers grow specific and meaningful. So deeper networks see complex shapes by stacking. Simple features become parts, parts become objects, one layer at a time.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the final layer's job? Let's try to understand. In a simple way. By now the hidden layers have done their job, turning the raw input into useful features.

One layer is still left at the very end: the output layer. Its size is special: one neuron for every possible answer. Sorting photos into cat, dog, or bird? Then the final layer has three neurons, one per choice.

Each of those neurons does the usual weighing and puts out one number: a raw score for that answer. Higher score, stronger lean toward it. So the output layer hands you a short list of scores, one per answer. Cat gets two point one.

Dog gets five point four. Bird gets zero point three. Now the decision is simple. Look for the biggest number in that list.

Whichever answer owns the highest score is the network's pick. Here, dog wins. These scores are still raw. They are not tidy percentages yet.

But to name the winner, you do not need that. The order alone is enough. The shape of this layer matches the task. One neuron for a yes or no question.

Ten for the ten digits. Thousands when the answer is a word. So the final layer is the readout. Everything the network noticed flows in as features and comes out as one clear choice.

That is where the answer appears. So the final layer holds one neuron per answer. Each gives a score, and the biggest score wins. That single comparison is the network's decision.

Quick check now. One question is coming up. Let's see if it clicked.

How does turning scores into probabilities work? Let's try to understand. In a simple way. The final layer hands you a raw score for each answer.

Just numbers. Cat two point one, dog five point four, bird zero point three. But these are not probabilities. You cannot say the model is seventy percent sure.

They can be negative, and they never add up to a whole. What we want is each answer as a percentage. Every value positive, and all of them adding to one hundred percent. That is a probability.

Step one makes every score positive. Raise a fixed number e to the power of each score. Negatives become tiny, bigger scores climb fast. Because that growth is so fast, it widens the gaps.

A small lead in the raw score becomes a big one here. Step two shares them out. Add these positive values into a total, then divide each by it. Every answer becomes a slice, adding to one.

These two steps together are called softmax. Soft, because it does not give everything to the winner. It spreads smooth probabilities while still favouring the top. That spread shows confidence.

Scores far apart give one answer near one hundred percent. Scores close together stay spread out, a hedged guess. And notice the biggest score still wins the biggest probability. Softmax never changes the winner, it just turns the ranking into readable numbers.

So softmax takes the raw scores, makes them positive, and shares them into probabilities that add to one. Same winner, now with confidence. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a loss function? Let's try to understand. In a simple way. The network has just made its guess.

Cat seventy percent, dog twenty, bird ten. It is leaning toward cat. But this example came with a correct label. The true answer here is dog.

So that confident lean toward cat is a mistake. We need to grade that mistake. The loss function takes the guess and the true answer, and returns one single number. How wrong you were.

The scale is simple. Put all your confidence on the right answer and the loss sits near zero. The further off you are, the bigger it grows. And it cares about confidence.

Being ninety nine percent sure of the wrong answer earns a huge loss. A careful hedge is punished far less. Notice what happened. A whole list of outputs, however long, is crushed into one lonely number.

That is the entire point. And that one number is the target. Training has a single job, make the loss smaller. Lower loss means a better network.

For choosing a class, the usual loss is cross entropy. It looks at the probability you gave the true answer, and rewards you for making it high. So the loss function compares the guess to the true answer, and boils the gap into one number. Zero is perfect, and bigger means more wrong.

Quick check now. One question is coming up. Let's see if it clicked.

How does a network find the direction of improvement? Let's try to understand. In a simple way. Training starts with a loss, one number saying how wrong the network is right now.

The whole goal is to make that number smaller. But the network has millions of weights. Each one could nudge up or nudge down. Which changes actually shrink the loss?

That is the real question. Take a single weight. Nudge it up a hair and watch the loss. If the loss rises, that was the wrong way.

If it falls, you improved. That tiny test has a name. It is a slope. How much the loss moves when the weight changes a little.

A steep slope means this weight matters a lot. Picture the loss as a hilly landscape. Your current weights are one spot on it. Lower ground means less wrong.

So improvement just means walking downhill. Now do that slope for every weight at once. Bundle them all together and you get the gradient. One arrow pointing toward higher loss, the steepest way up.

The direction of improvement is simply the opposite. Flip that arrow around. Stepping against the gradient takes the loss down as fast as possible. And the gradient is per weight.

A big slope says move this one boldly. A slope near zero says this weight barely matters, so leave it alone. So the gradient is a slope for every weight, packed into one direction. It points uphill toward more loss, so the network steps the other way to improve.

Quick check now. One question is coming up. Let's see if it clicked.

How does backpropagation work? Let's try to understand. In a simple way. The network already ran forward and made its guess.

The loss graded it, one number for how wrong it was. To fix that, every weight needs its own slope. Which way, and how hard, should this one move to shrink the loss? And there are millions of weights.

You could nudge one weight, run the whole network again, and check the loss. But doing that millions of times would take forever. Backpropagation is the shortcut. It starts at the output, where we already know the error.

How far the guess sat from the truth. Then it passes that error backward, one layer at a time. Each layer gets a note saying how much it added to the mistake. A neuron reads its note, finds the slope for its own weights, then writes a fresh note for the neurons feeding it.

The notes are built by multiplying. Each layer scales the signal coming back by its own local effect. Chained together, the blame reaches every weight. That is the trick.

One backward sweep, from output to input, hands every weight its slope at once. No re-running for each one. That is why it is called backpropagation. The error travels backward, opposite the forward pass, from the answer down to the first weights.

So backprop sends the error backward, and each weight learns its share of the blame. Out comes the full gradient, ready for the next step. Quick check now. One question is coming up.

Let's see if it clicked.

How does the learning rate work? Let's try to understand. In a simple way. By now the network knows which way each weight should move.

But one thing is missing. How far to step. Each weight update looks the same. Take its slope, multiply by one small number, then subtract.

That small number is the learning rate. It is a single dial you set before training. One number that scales every step the network takes. Set it too big and the steps overshoot.

The network leaps past the bottom, and the loss bounces around instead of settling. Set it too small and the network barely moves. Every step is a crawl. Reaching the bottom could take forever, if it ever arrives.

A good rate sits in between. Big enough to make real progress, small enough to stay on track. The loss slides down smoothly, step after step. Picture rolling into a valley.

The direction is fixed, downhill. The learning rate is your stride length. Long strides fly past the low point. Tiny strides take an age.

Here is the catch. The network does not learn this number. You pick it by hand, and try again when it is wrong. It is a dial worth tuning.

In practice it is tiny. Often around zero point zero one. Small steps, taken many thousands of times, add up to a trained network. So the learning rate is the size of each step downhill.

Too big overshoots, too small crawls, and the right value learns fast without falling apart. Quick check now. One question is coming up. Let's see if it clicked.

How does a single example update the weights? Let's try to understand. In a simple way. Start right after one training example has gone through the network.

Two things came out. A loss, saying how wrong it was, and a slope for every weight. The update is one simple rule, applied to each weight. Take its old value.

Subtract the slope times a small step size. That gives the new value. Watch one weight. It sits at zero point eight, with a slope of plus two.

Multiply the slope by the step, zero point zero one, giving zero point zero two. Subtract, and it becomes zero point seven eight. Now the twist. This same rule fires for every weight at once.

From this one example, the whole network shifts, each weight nudged a tiny amount by its own slope. And a single example is a rough guide. It pulls the weights to do better on itself, which can be slightly wrong for other examples. One nudge is noisy.

So you do it again. Next example, another tiny update. Then the next. Thousands of small, noisy nudges pile up, their errors cancel, and real improvement is left behind.

Updating from one example at a time has a name. Stochastic gradient descent. It is cheap and fast, since you never wait to see all your data before you step. So one example updates the weights like this.

Read each slope, subtract a small multiple, and every weight shifts a hair. One tiny step, then on to the next example. Quick check now. One question is coming up.

Let's see if it clicked.

How does one training step work? Let's try to understand. In a simple way. One training step is a short loop with four stages.

A forward pass, then a loss, then a backward pass, then an update. Every step runs all four, in that order. Stage one, the forward pass. An example enters the input, flows layer by layer, and the network puts out a prediction.

Just a guess, nothing has changed yet. Stage two, the loss. Compare that prediction to the true answer. The gap becomes a single number, the loss, saying how wrong this guess was.

Stage three, the backward pass. Send that error backward through the network. Out of it comes a slope for every weight, which way each one should move. Stage four, the update.

Nudge each weight against its slope, by a small step size. That is the only stage where the weights actually change. Those four stages, run once, are one training step. One example goes in, and the whole network shifts a hair to fit it a little better.

Then the loop starts over. A new example, another forward pass, another tiny update. Millions of these steps are what we call training. And the order is fixed.

You cannot update before you have slopes, cannot get slopes before a loss, cannot get a loss before a prediction. Forward, then backward, always. So one training step is this loop. Predict, measure the loss, pass the error back, update the weights.

Run it again and again, and the network learns. Quick check now. One question is coming up. Let's see if it clicked.