How to read AI papers

What exactly is a paper's one big claim? Let's try to understand. In a simple way. A paper can run twenty pages.

But underneath all the tables and equations, the authors are betting on one sentence. It is usually hiding in plain sight. Read the last two lines of the abstract. Phrases like, we show that, point straight at it.

And it gets repeated. The title squeezes it into a few words. The end of the introduction states it in full. The conclusion says it again.

A real claim has three parts. The thing they built. What it does. And a comparison.

Better than what, measured how. Here is one, in plain words. Our model translates more accurately than the best previous system, and trains in half the time. All three parts are there.

A paper makes many smaller claims. Every table makes one. Those are supports. The big claim is what they are all holding up.

So here is the test. Cross a sentence out and ask, does the paper still stand? A support can go and it survives. Take the big claim out, and the whole thing falls over.

Watch the edges too. On which data, at which size, against which baseline. Those words are the fence around the claim. Then write it out yourself.

One sentence, your own words. If you cannot, you have not read the paper yet. So, the one big claim is the sentence the paper cannot live without. Find it, check its parts, then read everything else as evidence.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the problem a paper is solving? Let's try to understand. In a simple way. Every paper is an answer.

And an answer only makes sense once you know the question. So find the question first. The problem lives on the first page. Paragraph one sets the scene.

Paragraph two tells you what goes wrong. Look for the turn. However. Yet.

But. That one word is the hinge, where the paper stops describing the world and starts complaining about it. A real problem has three parts. What people do today.

What breaks when they do it. And who pays for that. Here is one. Older translation models read a sentence one word at a time.

Long sentences crawled, and training took days. So always ask what it costs. Slower. Less accurate.

More expensive. Or flatly impossible. If nobody can name the cost, the problem may not be real. Now a trap.

Sometimes the problem is just the absence of their method. Nobody has tried this yet, is not a problem. It is only an opening. Good problem statements carry a number.

Machines score sixty, people score ninety five. That gap is the size of the problem. Here is the test. Say the problem back without using their method's name.

If you cannot, you have learned their solution and skipped the question. So, the problem is the gap a paper wants to close. Find it on page one, and every choice later makes sense. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is a paper's competition? Let's try to understand. In a simple way. Every paper has rivals.

But the authors decide which rivals you get to meet. That choice is the frame. Rivals show up in two places. The related work section, written in prose.

And the baseline rows of the results table. The prose is polite. The table is the real fight. Whoever sits in those rows is who the authors chose to be measured against.

Related work sorts older methods into a few families. Then each family gets a however. Watch that sentence. It names a weakness their own method happens to fix.

So first, look for who is missing. If the strongest known method is not in the table, that absence is the loudest thing on the page. Second, check the dates. A baseline can be a real system, but an old version of it.

Beating last year's model is a smaller claim than it looks. Third, check the effort. If they tuned their own model for weeks and ran the baselines at default settings, the numbers are not comparing methods. They are comparing care.

Also check the budget. Same amount of compute, similar model size, same training data. A fair fight holds those equal and changes only the idea. Honest framing is easy to spot.

The strongest rival is present, tuned properly, and the paper admits where that rival still wins. So, read the baselines before the results. The competition a paper picks tells you how big the win really is. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is an architecture diagram? Let's try to understand. In a simple way. Almost every paper has one big figure.

That figure is the whole method in a single picture. It is not decoration. It is the map. Start at the edges.

Find where the data enters, usually on the left. Then find what comes out the other end. Every box in between is an operation. Something goes in, something different comes out.

So for each box, ask one question. What did this change? Arrows are not just lines. Each one carries a tensor, a block of numbers.

The little label beside it is that block's shape. A shape looks like this. Five hundred by seven sixty eight. Sequence length, then dimension.

Where two arrows meet, those numbers have to line up. Watch for a small n times printed on a block. That block repeats. One box on the page can be ninety six real layers.

When an arrow splits, the same numbers travel two ways. That is a skip connection. Two arrows merging means adding, or gluing them together. Now the shading.

Authors color their own new piece and leave the standard parts gray. So the colored box is the part you came to read. Some of the picture exists only during training. The loss at the end, the dashed arrows going backward.

When the model runs, that half is gone. So, follow the arrows, question every box, and check the shapes. Read the figure that way, and the text just confirms it. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is a paper's novel contribution? Let's try to understand. In a simple way. Almost everything in a paper is borrowed.

The architecture, the dataset, the training recipe. Only one or two pieces are actually new. Those new pieces are the contribution. Everything else is scaffolding, so the new part has somewhere to stand.

The authors do tell you. The end of the introduction has a bullet list, our contributions are. Read it. But do not trust it yet.

Because that list is padded. Releasing code is a service, not an idea. Look for the bullet that changes how something works. The verbs give it away.

We propose, we introduce, we replace. That part is new. We use, we follow, standard practice. That part is borrowed.

Here is the real test. Take their system and delete the new piece. If what is left is an ordinary model from last year, you have found it. The ablation table checks your answer.

Remove that piece and the score should fall hard. If it barely moves, the real contribution is somewhere else. Contributions come in kinds. A new mechanism.

A new dataset. Or a finding, where nothing is built and something is finally measured. One trap. Bigger is not new.

The same method with ten times the compute is a result, not an idea. Ask what would hold at small scale. So, say it in one sentence. This paper adds this one thing to what already existed.

If you cannot fill that blank, keep reading. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a downstream task? Let's try to understand. In a simple way. You will meet this sentence everywhere.

We evaluate on eleven downstream tasks. It sounds technical. It is actually simple. Modern papers train in two stages.

First a huge generic stage, on raw text or images, with no task in mind. That stage is called pretraining. Everything that comes after that flows downstream. So the real jobs you point the finished model at, those are the downstream tasks.

Sentiment labels on reviews. Answering questions. Naming the object in a photo. Small, specific, and each one has real labels.

Here is the point. The pretraining objective is never the goal. Guessing hidden words is a means. The downstream score is the evidence.

So how does the model get to the task? Three ways. Freeze it and train a tiny head on top. Fine tune every weight.

Or just prompt it. Always read which one they used. A frozen model that scores well has learned something general. A fully fine tuned one may just be relearning the task.

Ask how much labeled data the downstream task got. The full training set, or a hundred examples? Those are two very different claims. One trap.

The authors pick the downstream tasks. If every one of them looks like the pretraining data, you are seeing memory, not transfer. So, downstream just means after pretraining. The tasks the model is finally judged on.

Ask which tasks, how it was adapted, and how much data. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a useful benchmark? Let's try to understand. In a simple way. A benchmark is three things bolted together.

A fixed set of examples, a hidden answer key, and one number at the end. It exists for one reason. So two models can be measured with the same ruler. So what makes one useful.

First, room at the top. If the best model already scores ninety nine, the benchmark is finished. It cannot show progress. Second, it has to spread models apart.

If everyone lands within half a point, you are reading noise, not skill. Third, the answers must stay hidden. Many benchmarks were scraped from the web, and models read the web. A model that memorized the test looks brilliant and learned nothing.

Fourth, the score has to stand for something you care about. Multiple choice trivia is not the same skill as writing code that runs. Fifth, give the number a floor and a ceiling. Random guessing scores twenty five.

Careful humans reach ninety. Sixty means very different things between those two. And the sharpest test. Can a dumb rule beat it?

If always picking the longest option scores well, the benchmark is leaking the answer through its shape. So when a paper leans on a benchmark, go look it up. How old is it, and what is the top score today. A benchmark is useful when it has room to grow, separates models, hides its answers, and measures the thing you actually want.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a results table? Let's try to understand. In a simple way. A results table is just a scoreboard.

Each row is a method, each column is a test, and every cell is one score. Theirs is the last row. First, find which way is better. Accuracy goes up.

Error and perplexity go down. A tiny arrow beside the metric tells you. Then read the caption. Which dataset, which metric, and was this the held out test set, or the set they tuned on?

Now compare fairly. Look for a column of model sizes. If their row is three times bigger, that is more compute, not a better idea. Bold marks the best number.

But the authors chose the bolding. So measure the gap yourself. A two tenths lead is nothing. Look for a small dagger or a footnote.

It usually means that number was copied from another paper, not run here, under a different setup. Watch the dashes. An empty cell means that method was never run there. Ask why the comparison you most wanted is the missing one.

Be careful with an average column. One huge win on a single test can carry the average while the method quietly loses the other four. So read it backwards. Hunt for the column where their number is not the best.

What a paper does not bold is the honest part. So, rows are methods, columns are tests. Check the direction, check the sizes, check whose numbers those are, and check where they lose. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is an ablation study? Let's try to understand. In a simple way. An ablation study is a controlled removal.

Take the finished model, delete one piece, train again, and see how far the score falls. A new model is never one idea. A new gate, a new loss, a longer training run. The ablation asks which one did the work.

The drop is the answer. Remove the gate, the score falls four points, and that piece carries the paper. Falls by nothing, and it was decoration. For that to mean anything, only one thing may change per row.

Same data, same size, same budget. Change two, and you cannot say which hurt. Compare the drop to the win. If the paper beats everyone by one point, a row that moves half a point proves nothing.

Now look for the row that is not there. Authors ablate the parts that survive. If their proudest idea has no line, that silence is the finding. Ask if they retrained.

Switching a piece off at test time, without training again, breaks a model that grew around it, and exaggerates the drop. Removing one at a time also hides teamwork. Two parts can each look useless alone, while taking both away makes the model collapse. The stronger version runs the other way.

Start from the plain baseline, add one piece at a time, and watch where the number jumps. So, break it on purpose. The size of the drop, and the row nobody ran, is the real story. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is an error bar? Let's try to understand. In a simple way. A score in a paper is usually one training run.

Change the random seed, train it again, and the number moves. Sometimes by a lot. So honest papers run it several times. The dot is the typical score, and the bar around it shows how far that score wandered.

But a bar is not one thing. It can be a standard deviation, a standard error, or a confidence interval. Only the caption says which. Then ask how many runs.

Three seeds is common, and three is barely enough. The bar is itself a guess from three numbers. Now the test that matters. Put their bar beside the baseline's.

If the two ranges overlap heavily, the win has not been shown. Most tables carry no bars at all. Then every number is a single sample, and a small lead can be nothing but a lucky seed. Watch for bars on their row only.

The baselines are single numbers copied from older papers, so their spread was never measured. Two different wobbles hide here. One comes from retraining with new seeds. The other comes from the test set.

A small test set gives a wide bar. And beware the best of five. Report your luckiest seed against a rival's average, and the gap is measuring luck, not method. So, the dot is the guess and the bar is the doubt.

Ask what the bar means, how many runs, and whether the bars overlap. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is author speculation? Let's try to understand. In a simple way. Every sentence in a paper is one of two things.

Something the authors measured, or something they think it means. A result is a number an experiment produced. This model, this dataset, this metric, this score. Run it again and it comes back.

Speculation is the story around that number. Why it worked, what it means, what happens next. No experiment stands behind it. Mostly they live in different rooms.

Results and tables hold the numbers. Discussion and conclusion hold opinions. The abstract mixes both in one breath. So here is the test.

Point at any claim and ask which table produced it. If you cannot name one, you are reading an opinion. The commonest slip is claiming a mechanism from a score. Accuracy rose two points was measured.

The model learned to reason was not. A score never says why. Then watch the scope. They tested one dataset, one model size, one language.

The sentence says language models, in general. The number's fence is much narrower. A simple defense. Read the tables before the prose.

Form your own reading of the numbers, then let the authors tell you theirs. Speculation is not the enemy. It is where the next paper starts. The problem is only when it walks into the abstract dressed as a finding.

So, sort every sentence. Measured, or interpreted. The tables are the paper's evidence. Everything else is the authors thinking out loud.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a weasel word? Let's try to understand. In a simple way. Here are two claims.

Our method may improve accuracy on some tasks. And, our method adds one point two accuracy points on ImageNet. Only the second one can be checked. A weasel word is a word that quietly removes the testable part of a claim.

Cross it out, and see whether anything checkable is left. The first family is hedges. May, can, could, suggests. A hedged claim survives any result, so it was never really a prediction.

Next, the vague size. Significantly better. Substantially faster. Neither one is a number, so always ask, better by how much.

Then the vague scope. On several benchmarks. In many settings. Ask which ones, and ask how many were run and never shown.

Up to is the sneakiest. Up to three times faster is one best case, on one lucky setup. The typical case is missing. Watch for claims with nobody behind them.

It is widely believed. It has been shown. If no citation follows, nobody has shown anything. Comparable to, and competitive with, usually mean slightly worse.

If their model actually won, the sentence would say so with a number. Now the real skill. Hold the claim next to the evidence. Careful words over a solid table is modesty.

Bold words over a thin table is spin. So rewrite the claim with numbers in it. How much, on what, compared to what. If you cannot fill those in, there is no claim yet.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a state of the art claim? Let's try to understand. In a simple way. State of the art means, the best result anyone has published.

But said on its own, that phrase is not finished. Best at what? A real claim names three things. The task, the exact dataset and split, and the metric.

Missing any one, and there is nothing to check. It also has a date. State of the art is a claim about one day. Papers freeze at submission, so last year's best is often third today.

Now watch the qualifiers. Best among open models, under seven billion parameters, on the low resource split. Every extra word shrinks the field they had to beat. Then find the runner up.

The row above theirs should be the strongest published rival. If the obvious rival is missing, that absence is the result. Ask how hard each side was tuned. Authors polish their own model for months, then run the rival at default settings.

That alone can look like a win. And ask how the winner was chosen. Trying twenty versions and reporting the best one on the test set is choosing after seeing the answer key. The check itself takes two minutes.

Open the leaderboard for that dataset, sort by that metric, and find their number. Then look at the date. So, state of the art is a number, on one split, with one metric, on one day. Ask what shrank the field, and who is missing.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a paper's code release? Let's try to understand. In a simple way. A paper describes what was done, written up months later.

The code is what actually ran. Where the two disagree, the code is right. So find it first. Usually a link in the abstract, or a footnote on page one.

If not, search the paper title on GitHub. Then check the promise. Code will be released soon is not code. A repository holding a readme and nothing else is a plan, not a release.

Look at what is there. Code to run the finished model, or code to train it? Weights alone let you use it, not rebuild it. Open the config file.

Learning rate, batch size, number of steps, warmup. Those numbers are often nowhere in the paper. And there sit the small extras. A gradient clip, one more augmentation, a schedule longer than the paper said.

Those quiet lines often carry the gain. Now open the issues tab. That is a public reproduction report. If ten people say table three will not reproduce, you know something the paper never says.

You never have to run it. Reading the model file settles what the diagram left vague. Where the norm sits, what the loss really is. And no code at all?

That is not proof of anything wrong. But the result now rests on their word, so hold it loosely. So, the paper is the claim. The code is the record.

Find it, open the config, read the issues. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the five question test for a paper? Let's try to understand. In a simple way. A paper is thirty pages and you have twenty minutes.

Reading it front to back is the slowest way in. What you want is a fixed set of questions. The same five questions, in the same order, for every paper you open. Not a summary.

A short interrogation. One. What is the claim? One sentence, in your own words.

If you cannot say it out loud, you have not found it yet. Two. What holds it up? Point at the table or the figure.

A claim with nothing under it is just a sentence. Three. Compared to what? Every number needs the thing it beat sitting right beside it, or it means nothing.

Four. Where does it stop working? One dataset, one model size, one language. That fence is the real size of the finding.

Five. Could you run it yourself? Weights, code, the exact settings. If nobody can repeat it, it stays a story.

The order matters. The claim comes first, because the other four all point back at it. Answer them out of order and you drift. And you are allowed to stop.

If question two has no answer after five minutes, close the tab. The checklist is also a filter. So, five questions. The claim, the evidence, the comparison, the limits, and whether you could check it.

Write the five answers down and you have read the paper. Quick check now. One question is coming up. Let's see if it clicked.