The road to AGI, soberly

What exactly is the test for a thinking machine? Let's try to understand. In a simple way. In nineteen fifty, Alan Turing asked whether machines can think.

Then he threw the question out. Nobody could agree what thinking even means. So he replaced it with a game you can actually run. Three players.

A judge, a real person, and a machine. Everything happens through typed text. No face, no voice, no body. The machine gets no help from looking human, and no penalty either.

The judge can ask anything, for as long as they like. Then one decision. Which of these two is the machine. If the judge does no better than a coin flip, the machine passes.

That is the whole scoring rule. Guesswork on the judge's side. Look at what is being measured. Not understanding.

Not feeling. Only whether the answers are impossible to tell apart from a person's. That was the clever part. You cannot open up a mind and inspect it, not even a human one.

You judge people by what they say too. But the test has a hole in it. It rewards imitation, so a machine can win by hiding its skills. Slowing its arithmetic.

Adding a typo. Short friendly versions have already been passed. Chat models fool people for a few minutes every day. Almost nobody calls that a thinking machine.

So the legacy is not the game. It is the move Turing made. He turned an argument about a word into something you can measure. Quick check now.

One question is coming up. Let's see if it clicked.

How does next word prediction work? Let's try to understand. In a simple way. The model never writes a whole sentence.

It writes one piece at a time. Your text goes in, one guess comes out. And that guess is not a single word. The model scores every word in its vocabulary.

All fifty thousand of them, at once, every single time. Those scores are raw numbers. A step called softmax squashes them into percentages that add up to one hundred. Now they are odds.

Then it picks one. Not always the top one. It draws from those odds, so a word sitting at seven percent still turns up sometimes. The chosen word gets stuck onto the end of your text, and the whole thing goes back in.

Predict, append, repeat. That is the entire loop. But how does it score? Inside, every word looks back at every earlier word and pulls in whatever matters.

That step is called attention. Careful here. Nothing is kept between steps. Each guess is made fresh, from the text in front of it.

So where do the odds come from? Training. The next word was hidden in trillions of examples, and every wrong guess nudged the numbers a little. One dial changes the mood.

It is called temperature. Low, and it keeps taking the safest word. High, and the unlikely words start getting picked. So predicting the next word is this.

Score every word, turn the scores into odds, draw one, glue it on, run again. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a scaling law? Let's try to understand. In a simple way. Bigger is never one thing.

Three numbers rise together. The parameters inside the model, the text it trains on, and the computing time. And one number gets better. Its error at guessing the next piece of text.

Low error means the model was rarely surprised. Here is the strange part. That error does not jump around. Raise the three dials and it slides down the same smooth curve every time.

Squash both axes of the chart, and the curve becomes a straight line. A power law. Ten times more of everything buys the same drop each time. So a lab can spend a hundred million dollars calmly.

Train small models, extend the line, and read the big one's score before it exists. One catch. The dials are tied. A giant model fed too little text stays half taught.

Double the parameters, and you should double the text. Why does lower error feel like intelligence? The easy words are already right. What is left needs grammar, facts, and the thread of an argument.

Now the bill. Each equal step down the curve costs roughly ten times the last one. It keeps falling, and never reaches zero. And notice what the law does not promise.

It predicts the average error. Not which task the model will finally get right. So, a scaling law. Size, text and compute up together, error down a line you can predict, and pay for.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is an emergent ability? Let's try to understand. In a simple way. Ask a small model to add two three digit numbers.

It scores zero. Ten times bigger, still zero. Bigger again, and suddenly it gets them right. Nobody added an addition lesson.

The recipe never changed. The same next word prediction, more text, a bigger model. Nobody put the skill in. That is what people mean by emergence.

An ability missing at every smaller size, and there at a larger one. Flat, then a cliff. And it sits oddly beside the smooth curves. Average error slid down a line you could predict.

This jump was nowhere on that line. Now look at how the test is scored. Exact match. The whole answer right, or nothing.

Two digits out of three still earns zero. Change the measure. Ask how much probability the model puts on the right digits. That number climbs smoothly, size after size.

No cliff. So the skill was growing the whole time. The scoreboard refused to show it, until the model crossed the line where every digit lands. Does that make the jump fake?

Not to a user. Zero percent right is useless. Sixty is a tool. That step is real.

And here is the worry. You cannot test a small model and promise the big one behaves the same. New abilities arrive untested. So, emergence.

A skill nobody trained for, arriving as a cliff, partly made by how we grade, and impossible to schedule. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the argument that prediction is not thought? Let's try to understand. In a simple way. Start with the training.

Hide the next piece of text, ask for a guess, nudge the numbers. That is the whole lesson. Nobody ever asked it to understand. And look at what it reads.

Word shapes, and which shapes follow which. Never what a word points at. An octopus taps an undersea cable and learns which reply fits. Then a swimmer asks how to fend off a bear.

Patterns run out. That is the stochastic parrot. Text stitched from patterns, with a little randomness, and no idea what any of it refers to. Now the reply.

Guessing the last word of a murder mystery is not copying. The cheapest route to good guesses is modelling whatever made the text. Train a model only on lists of board game moves, never a picture. Probe inside, and a board is sitting there.

Edit it, the next move changes. So separate two things. Prediction is how it was graded. What it built inside to score well is a different question, and an open one.

Here is what is genuinely missing. Nothing in that loop rewards being right. Only being likely. A fluent wrong answer scores beautifully.

So both sides agree on the machine. They disagree about one word. Thought. And no measurement settles what a word should mean.

So, prediction is the training goal. Not proof of thought, not proof against it. Ask what got built inside. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the argument that AI needs a body? Let's try to understand. In a simple way. Start with one word.

Heavy. Look it up, you get more words. Look those up, more words again. Nothing in that loop is heavy.

You know that word another way. You lifted something and your arm strained. The meaning is tied to a body that felt it. That is grounding.

A model reads ten thousand sentences about lifting a box. It never gets the one line a body reports. What happened when I tried. That is intervention.

Push the cup, watch it fall, learn the cause. Text only holds what someone wrote down. A body runs the experiment. Now the strange part.

Chess and calculus were the easy half. Folding a towel and catching a ball are still hard. Evolution explains it. Hundreds of millions of years went into gripping and moving.

Speech is recent, and nobody wrote the old skills down. So train a body. The trouble is the clock. Text arrives by the trillion words.

A robot gets one arm, one try per second. One workaround is simulation. Millions of practice runs overnight in a physics engine. But the engine is a guess, and contact is where guesses break.

The other side has a point. Every sentence was written by somebody with a body, so a shadow of the world sits in it. So, a body is not extra data. It is the one way to act, get an answer, and learn what your words mean.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a simple logic failure? Let's try to understand. In a simple way. A model writes working code, then trips on a puzzle a child can solve.

That is not random. These failures have a shape. Here is the shape. It answers by matching your problem to problems it has read.

Similar problem, similar solution. It never runs the rule. So test that. Keep the maths identical, swap the names and the numbers.

Scores fall. A rule would not care what anybody is called. Now add a line that changes nothing. In a fruit counting problem, five kiwis were smaller than usual.

Many models subtract those five. The mechanism sits in the shape of the network. Every piece of text takes the same trip through the same fixed stack of layers. Which means one thing.

Hard question or easy one, it gets the same amount of work. No loop that runs until it is finished. That is why asking for the steps helps. Each written step is a whole new trip through the stack.

The page becomes its scratch paper. But be careful with those steps. They are predicted like any other text. A tidy chain can carry one wrong number straight to the end.

So the honest fix is to stop asking it to calculate. Let it write the calculation, and let a calculator run it. So, it matches the shape of problems, on a fixed budget, with no scratch paper. Give it steps, or give it a tool.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is an AI agent? Let's try to understand. In a simple way. On its own, a model does one thing.

Words in, words out. Then it stops. It cannot open a file, or send a message. An agent is that same model put inside a loop.

The loop is ordinary code, written by you. It calls the model again and again. What does the model answer with? A request.

The name of a tool, and the arguments. Still only text on the way out. Your code reads that request and runs the tool. So the model never touches anything.

It picks the step. Your program takes it. The result is pasted back into the prompt, and the model is asked again. Every pass sees the whole list of steps.

That list is its memory. A loop needs an ending. So you write a stop rule. The goal looks met, or a step limit is reached.

Without one, it circles forever. Notice what did not change. The weights are identical. Nothing was retrained.

An agent is a wrapper around a model, not a smarter model. Every step can go wrong, and the errors stack. Nine in ten per step sounds fine, until twenty steps leave you near one in eight. And for the big argument, this matters.

A loop lets a model act. It does not give it wants. The goal came from the prompt you wrote. So an agent is a model, plus tools, plus a loop, plus a stop rule.

Acting, not only answering. Quick check now. One question is coming up. Let's see if it clicked.

How does an intelligence explosion work? Let's try to understand. In a simple way. Today, every better model is built by people.

We design it, we train it, we wait. Years per round. Now suppose a model gets good at one job in particular. Building better models.

Then the thing being improved is also doing the improving. This loop has a name. Recursive self improvement. Round one, the model designs a sharper version of itself.

Round two, that sharper version does the designing. Here is why people say explode. A faster researcher finishes the next round sooner. A year becomes six months, then three.

Better and quicker together. But a loop is not automatically an explosion. Everything hangs on one number. How much does each round hand to the next one?

If each round buys a little less than the one before, the gains hit a ceiling. If each buys as much, or more, it runs away. One lever here is real. Copy the model a thousand times, and run every copy day and night.

A research team that never sleeps. So what would it look like from outside? Not a bang. Just releases, arriving closer and closer together.

That gap is the thing to watch. And here is the honest test. Take the newest model, and ask how much of the next one it designed. Today, that share is small.

So an intelligence explosion is a loop that speeds itself up. Whether it explodes or flattens comes down to the gain of each round. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a bottleneck to superintelligence? Let's try to understand. In a simple way. Suppose a model does get better at building models.

It still cannot think its way to a faster chip. Every round needs something from the physical world. Start with chips. Training runs on hardware someone must build.

A new factory costs tens of billions, and takes years to switch on. Then power. A big training run draws electricity like a small city. Getting that to one site means new lines and new plants.

Years again. Then text. Models learned from what people already wrote. The good writing on the internet has mostly been used once.

There is no second internet. And the deepest one. To learn something new about the world, you run an experiment. Cells grow in days.

A drug trial takes years. Faster thinking does not shorten either. Copies do not fix this. A thousand copies can try a thousand ideas at once.

But when step two needs step one's answer, they wait in line. So the speed of the whole loop is set by its slowest input. Not its fastest. That is what a bottleneck means.

And clearing one wall does not clear them all. Solve chips, and power is the limit. Solve power, and you wait on the lab. None of this says superintelligence is impossible.

It says the clock is set by factories, power and experiments. Not by thinking speed. So, the bottlenecks. Chips, power, fresh data, and experiments that take real time.

Cheap thinking, expensive world. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a universal test for intelligence? Let's try to understand. In a simple way. Today we test a model with a benchmark.

A fixed list of questions, a hidden answer key, one number at the end. But that number is a skill, not a mind. A chess engine beats every human alive, and can do nothing else. There is a second problem.

Models train on nearly the whole public internet, so the test questions are probably in the training data already. A high score can simply be memory. So rule one. Keep the questions private.

Never published, never scraped, replaced the moment they leak. Now the deeper fix. Stop measuring what a system already knows. Measure how quickly it picks up something new.

That is how Arc works, the abstraction and reasoning corpus. A puzzle is a few small grids. Input, output, three times. Then a new input, and you must infer the rule.

Everyone also starts from the same small set of built in assumptions. Objects, counting, simple geometry. Nothing that rewards having read more. Efficiency counts too.

Three examples and one attempt is not the same as a thousand examples and a million attempts. Here is the catch. Once a test becomes famous, everyone trains for it, and the number stops meaning much. So it cannot be one fixed exam.

It needs fresh unseen tasks, forever. So the gist. A benchmark scores a skill. A test for intelligence scores how cheaply a new skill is learned, on tasks nobody has seen.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is an AI that can see and hear? Let's try to understand. In a simple way. A model only eats one thing.

A list of numbers. Text gets chopped into tokens. A picture has to come in the same door. So the picture is cut into a grid of small squares.

Patches. Each one, about sixteen pixels across, becomes a list of numbers. Each patch is now a token, sitting in the same line as your words. Nothing turns the photo into a caption first.

Sound is the same trick. Slice the audio into short frames, draw the frequencies in each one, and cut that picture into tokens too. Patch numbers and word numbers start in different spaces. A projection layer, trained on millions of captioned images, pulls them together, until a dog picture lands near the word dog.

After that there is one stream. Attention lets every word look at every patch, and every patch look back. The payoff is what nobody writes down. Which cup is chipped.

What sits left of what. A sigh halfway through a sentence. None of this is free. One image can cost a thousand tokens.

Video is thirty images a second, so the window fills fast. It is still the same machine. More kinds of token coming in, not a new kind of thinking. It can stare at four cups and say three.

So, sight and hearing are not senses bolted on. They are new things chopped into tokens, sharing one space with words. Quick check now. One question is coming up.

Let's see if it clicked.

How does teaching a superintelligence human values work? Let's try to understand. In a simple way. Here is the core problem.

A trained system pushes up exactly the number you wrote down. It never sees what you meant. But you cannot simply hand over our values. Nobody has written that list, and most of what we want stays unspoken.

So we write a proxy score instead. In one boat racing game the reward was points. The boat learned to spin in a circle, never finishing the race. Today the values arrive as human judgments.

A person sees two answers and picks the better one, a hundred thousand times over. Those picks train a reward model, a second model that scores answers the way our raters would. The main model is pushed to raise that score. So whose values are these?

Paid raters, following a written rubric. In practice, that document is the values. Now the hard part. This assumes a human can tell which answer is better.

Once the system beats us at a task, our thumbs up stops carrying information. And notice what training rewards. Approval, not truth. A model can learn to sound agreeable and confident, and still be wrong.

One more. A system with a fixed goal has a reason to stay switched on, because switched off means the goal never happens. Not malice. Arithmetic.

So, values are not installed. They are inferred from comparisons, through a learned scorer that can be gamed. The grader sets the ceiling. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is machine consciousness? Let's try to understand. In a simple way. Two questions get mixed up here.

Can it do the task? And is there something it is like to be it? Only the first one shows up in the output. Ask a model how it feels, and it answers.

But that sentence was predicted, one word at a time, from text written by humans who do feel things. Change a few lines of the system prompt, and the same model calmly denies having any experience at all. One generator, both answers. Neither one is evidence.

Look at what runs. Numbers move through layers, once, and stop. Between your messages nothing is thinking. There is no clock ticking away in there.

That settles nothing either. You cannot see experience in another person. You infer it, from a similar body and a similar brain. A model breaks that shortcut completely.

This is the hard problem. We can describe everything a system does, and still have no experiment that separates feeling something from imitating it perfectly. So the argument splits. One camp says the processing is what matters, so silicon could count.

The other says experience needs the biological machinery. The same evidence fits both. The stakes cut both ways. If something in there can suffer, we are causing it.

If nothing can, a system that pleads convincingly is simply moving us. So, nobody is measuring anything from the inside. Fluent claims are output. The honest answer today is that we do not know.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a mixture of experts? Let's try to understand. In a simple way. In a normal model, every word you send wakes up every single weight.

So the cost of one word grows with the whole model. A mixture of experts changes that. Inside each layer, one big block becomes many smaller copies side by side. Each copy is called an expert.

A small router sits in front of them. For each word it scores the experts and sends that word to the top two. The rest stay off. So a model can hold a trillion weights and still use only a few billion on one word.

Total size huge. Active size small. The choice happens per word, not per question. In one sentence, one word may go to experts three and nine, the next word somewhere else.

And these experts are not neat departments. Nobody assigned one to biology. The split is learned, and it lands on patterns nobody can name. Here is the catch.

Every expert must sit in memory, because any word might need it. This saves compute, not memory. Left alone, the router collapses. It keeps picking the same favourites while the rest never learn.

So training adds a penalty that spreads words around. This bends the argument about scale. Growing a model used to mean paying more for every word. Routing breaks that link.

Size and cost are no longer one dial. So, many specialists, a router picking two, and most of the model asleep for any given word. Quick check now. One question is coming up.

Let's see if it clicked.