Data: fuel of the machine

How does reading the whole internet work? Let's try to understand. In a simple way. A big model needs trillions of words.

Nobody could type that. But the web already wrote it down. Billions of pages, sitting there. So just download it?

There is no list of every page. Nobody keeps one. The web is only pages pointing at other pages. So machines walk it.

Start with a few known pages. Fetch one, read every link on it, add those links to a queue. Then fetch the next. That is a crawler.

The queue has a name. The frontier. And it grows faster than you empty it, because one page hands you a hundred more. Every site keeps a rules file called robots dot text.

It says what a crawler may take. Polite crawlers read it first, and wait between visits. What comes back is not clean prose. It is menus, ads, cookie banners and footers.

A second step strips that away and keeps the real text. And it never finishes. Pages change while you crawl. So a crawl is a snapshot, a dated photo of the web, running to petabytes.

One catch. Much of the web sits behind logins, paywalls and apps. A crawler never sees those. The whole internet means the public part.

And what you have now is a pile, not a dataset. The same page a thousand times. Junk. Cleaning it is its own job.

So, machines read the internet by walking its links. Fetch, extract, queue, repeat, until the snapshot is full. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a public data repository? Let's try to understand. In a simple way. You need data.

Your instinct is to go collect it yourself. Stop. For most tasks, somebody already collected it, cleaned it, and published it for free. These places have names.

Hugging Face, Kaggle, Zenodo, and government portals like data dot gov. Hundreds of thousands of ready datasets. A repository is not a folder of files. It is a searchable catalog.

You filter by task, by language, by size, by license. Every dataset carries a card. Who built it, where the rows came from, how big it is, and what you may do with it. And the data is already packaged.

Rows, columns, one fixed name. Two lines of code and it is in memory. Now the traps. Free to download is not free to use.

Many are marked research only. That line decides if your project is legal. Next, provenance. A card is a claim, not a proof.

Plenty of datasets were scraped from somewhere else, relabeled, and uploaded again. Trace it back. The sneakiest one. If a famous test set sat in public for years, the crawlers already swallowed it.

Your model may have seen the answers, so a high score means nothing. And the biggest pile is rarely the best. Ten thousand rows that match your task beat a million rows of something close but wrong. So, search the catalog, read the card, check the license and the source.

The treasure is real. It is just never labeled treasure. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is duplicate data? Let's try to understand. In a simple way. Crawl the web and you do not get millions of different pages.

One story lands on hundreds of sites. Legal notices sit on every one. Training cannot tell. It reads examples one by one, nudging the weights each time.

A paragraph sitting there a thousand times pushes a thousand times. Past a few dozen copies, something breaks. The model stops learning the pattern and memorizes the string. Prompt it right, and it recites the text back.

So we hunt them down. Run every document through a hash function. Out comes a short fingerprint. Same fingerprint, same document, drop it.

Change one word, or one date, and the fingerprint is completely different. Near copies stroll straight through. And they are most of them. So stop hashing whole documents.

Chop each into overlapping runs of a few words. Those are shingles. A document becomes a bag of them. Now similarity has a number.

Share ninety percent of your shingles, you are a near duplicate. Share five percent, you are another page. Comparing every pair of a billion documents is hopeless. So MinHash keeps a few smallest hashes per document.

A tiny signature. Lookalikes land in one bucket, and only those get compared. Then you pick a line. Eighty percent overlap and out it goes.

Too tight, and you delete honest pages that just share a phrase. Deduplicated, the pile shrinks and the model improves. Less memorizing, less rereading, more of the world for the same budget. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is a missing value? Let's try to understand. In a simple way. Every real dataset has holes.

A sensor that died. A form nobody filled in. That empty cell is a missing value. A model cannot multiply a blank.

Feed one in and training stops with an error. Somebody has to decide what goes there. The easy fix is to delete every row with a blank. Fine when holes are rare.

But if six columns each lose five percent, a quarter of your data is gone. So instead you fill the hole with a guess. That is called imputation. The row survives, and the column keeps a number.

The usual guess is the column's own average. For a lopsided column like income, use the median, because a few huge values drag the average. Fill a third of a column with one repeated number and you build a spike at the middle. The spread shrinks.

A guess is not a measurement. So keep a receipt. Add one more column that says, this cell was blank. Now the model can learn from the fact that it was blank.

And ask why the hole is there. If high earners skip the income box, the blanks are not random. They mark the rich. The average erases that.

One last trap. Compute that average on the training rows only. Average the whole table and your test rows quietly help fill themselves in. So, look at the hole, ask why it is there, fill it honestly, and flag that you did.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is feature scaling? Let's try to understand. In a simple way. Look at one row of a table.

Age, thirty four. Income, eighty two thousand. Rooms, three. Same row, wildly different ranges.

The model does not know what a year or a rupee is. It just does arithmetic. So when it measures how far apart two rows are, income shifts by thousands, age by ones. Income wins.

So we rescale each column on its own. The first recipe is min max scaling. Subtract the smallest value, divide by the range. Now the column runs from zero to one.

The second recipe is standardization. Subtract the column's average, divide by its typical spread. Zero now means average, one means one spread above. Which one?

Min max gives a neat fixed box. But one billionaire pushes everybody else into a tiny corner near zero. Standardization shrugs off outliers. Scaling also speeds up training.

With lopsided columns the loss surface is a long narrow valley, and gradient descent zigzags. Even scales round it out, so the steps head straight down. One rule matters. Measure the minimum, the maximum, the average and the spread from your training rows only.

Then reuse those exact numbers on every new row. Not everything needs this. A decision tree only asks which value is bigger, so scale never bothers it. But distances and weighted sums care.

Nearest neighbours, k means, neural nets. So, feature scaling puts every column in the same range. The loud ones stop shouting down the quiet ones. Quick check now.

One question is coming up. Let's see if it clicked.

How does image labeling work? Let's try to understand. In a simple way. Open a photo and the computer sees a grid of numbers.

Brightness values. Nothing there says cat. The answer must come from outside. So a person opens the image in a labeling tool, looks, and records an answer.

That answer is the label, and the act is called annotation. The task decides the shape of the label. For classification, the whole image gets one word. Cat.

That is all of it. Ask where, and the label becomes a box. Four numbers. Left, top, width, height.

Ask which pixels, and every pixel gets its own class. A mask. The annotator does not type freely. They pick from a fixed list of allowed classes.

That list is the schema, agreed before labeling begins. Real photos are awkward. Half a tail behind a sofa. A cat on a poster.

A written guideline settles those edge cases, so the same photo gets the same answer. The label never touches the pixels. It lives in a separate file: one line of JSON, the file name plus the answer. The name joins them.

Then training turns the word into a number. Cat becomes class two, a plain index into that list. A box is already four numbers. Shape also sets the price.

A one word tag takes a second. A box, about ten. A pixel mask can take minutes per image. So, a label is a human answer recorded beside an image, in the shape the task needs.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is inter annotator agreement? Let's try to understand. In a simple way. One person opens a photo and types a label.

But people get tired. They read the rules their own way. Ask again next week, and the answer can flip. So send the same item to three people, working separately.

Nobody sees the others' answers. Now one photo comes back with three opinions. And they disagree. Two say cat.

One says dog. That gap is not noise to sweep away. It is a measurement. So count it.

Out of one hundred items, how many did all three match on? Say seventy eight. That is your raw agreement. But raw agreement flatters you.

If ninety in a hundred photos are cats, two lazy labelers who always type cat agree ninety percent of the time, and know nothing. So subtract the luck. Cohen's kappa measures how far you beat chance agreement, scaled so zero is a coin flip and one is perfect. Above point eight is solid.

Now pick one final label. Take the majority. Two cats beat one dog, so the item ships as cat. The odd vote is outvoted, not erased.

Ties, and items everyone fought over, go up to an expert. That pass is called adjudication, and its answer becomes the gold standard. Human agreement is also the ceiling. If people only agree eighty percent of the time, a model scoring ninety is fitting someone's noise.

So, ask several people, measure how much they truly agree, then vote. That is how one label earns trust. Quick check now. One question is coming up.

Let's see if it clicked.

How does weak supervision work? Let's try to understand. In a simple way. You have a million messages and no labels.

A person can tag maybe a thousand a day. Hand labeling will never catch up. So stop labeling items. Label with code.

If a message says wire transfer, call it spam. One rule, and the whole million is tagged in seconds. Each rule is called a labeling function. And it may stay quiet.

No matching words, no vote. It abstains, instead of guessing. One rule is thin, so you write forty. Keywords, patterns, a lookup list, even an old model.

Each one narrow, each one sloppy. Now look at one message. Three rules shout spam. One says clean.

The rest abstain. And there is no answer key to settle it. Here is the trick. Count how often each pair of rules agrees.

A rule that keeps agreeing with the crowd is probably accurate. A loner is probably junk. That scoring is the label model. Then it votes, weighted by trust.

The output is not spam or clean. It is a number. Eighty five percent spam. The doubt is kept.

Last step. Train an ordinary classifier on those soft labels. It reads the raw text, not your rules, so it catches spam no rule ever mentioned. One catch.

If your rules share a blind spot, they agree while all being wrong. So keep a small hand checked set. So, rules vote, agreement scores them, and a model learns past them all. Quick check now.

One question is coming up. Let's see if it clicked.

How does synthetic data work? Let's try to understand. In a simple way. Your car has driven a million miles.

It has never seen a couch fall off a truck. The case you need most is one you cannot collect. So stop collecting. Build it.

Open a driving simulator, drop a couch on the road, press render. That frame never happened. Now you have it. And here is the prize.

The engine placed that couch, so it knows the exact pixels. The label comes out with the picture. Free. Notice the order flipped.

Normally you find an example, then pay for a label. Here you pick the label first, and build an example to match. Which hands you a dial. Rain at night is one frame in a million out there.

In the simulator, ask for ten thousand. It works on records too. Learn the shape of real patient files, then emit fresh ones with the same patterns and no real person inside. Now the catch.

Simulated rain is not rain. Textures too clean, light too perfect. The model learns that world, then meets ours. That is the reality gap.

And a generator only makes what it knows. Whatever your simulator never modeled, your model never sees. Its blind spots become yours. Push further.

Train on your own output, round after round, and the rare cases fade first. Everything drifts to the average. That is model collapse. So, you invent the example, and the label arrives free.

Synthetic data buys the rare cases. Real data keeps it honest. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a synthetic face? Let's try to understand. In a simple way. You need ten thousand faces to train a model.

Real ones cost money, and each one is somebody's privacy. So you make people who do not exist. It is not cutting pieces out of your album and gluing them together. Nobody's nose gets reused.

Here is the trick. Every image is a long list of numbers, so it is one point in a space. Your whole album is a cloud of points. Those points are not scattered everywhere.

Faces clump into one region. So imagining a new face means picking a point inside that region where none of your photos sit. But how does it learn that region's shape? Two networks.

A generator that builds fakes from pure noise. And a judge, trained on real photos. Now they fight. The judge calls each image real or fake, and every catch is feedback the generator learns from.

A generative adversarial network. Millions of rounds later, the judge is guessing at fifty fifty. The fakes have landed inside the real cloud. One failure is famous.

The generator finds a single face that fools the judge, and makes it again and again. Ten thousand images, one person. That is mode collapse. And here is the ceiling.

It only imagines inside the cloud you showed it. No glasses in your album, and it will not dream them up. So, learn the region where faces live, then sample a fresh point inside it. New faces, no real person.

Quick check now. One question is coming up. Let's see if it clicked.

How does synthetic text work? Let's try to understand. In a simple way. You want a million worked examples in your exact format.

Step by step, in your tone. Nobody on the web ever wrote those. So you stop looking, and ask a model to write them. It is already fluent.

Give it a prompt, and out comes the example you needed. Start with a handful you wrote by hand. Show them, ask for more in the same shape, and the keepers become new seeds. That is self instruct.

Now the hard part. Fake text comes with no answer key. Fluent and wrong reads exactly like fluent and right. Most of it is junk.

So never trust one draft. Ask for twenty, then throw nineteen away. That is rejection sampling, and it is where the quality actually comes from. Throw away by what?

A checker. Code runs the unit test. Math checks the final number. Writing an answer is hard.

Checking one is cheap. Most tasks have no checker. Then a second model grades the draft against a rubric, or you keep what several drafts agree on. Now the payoff.

A big teacher writes the surviving examples. A small student trains on them, and matches it on that one task. One limit. A filter picks, it never invents.

If the generator cannot get it right even once in twenty, no filter will find a good answer. So, a model writes the drafts, a checker throws most of them away, and what survives is your training data. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is the garbage in garbage out problem? Let's try to understand. In a simple way. Here is one training row.

A photo of a husky, labeled cat. You spot that in a second. The model cannot. It has never seen the world.

Because training has one rule. Move the weights until the output matches the label. The label is never checked against reality. It is the definition of right.

So the wrong row is not ignored. It looks like a huge mistake, so it pulls the hardest. The model bends to please it. Still, one bad row is survivable.

Random slips point every which way, so they cancel out. What kills you is the same mistake, repeated. Say every blurry photo was rushed and labeled cat. That is not noise.

That is a pattern, and a model is built to find patterns. So blurry now means cat. So you reach for a bigger model. And it gets worse.

More capacity means it fits your data more faithfully. Garbage included. Power is not a filter. And here is the cruel part.

Your test set came from the same pile. So it scores ninety five percent, and you believe it. The ruler is bent too. So there is no fix downstream.

No loss function, no clever trick. Go back and read a random hundred rows by hand. Count the bad ones. That is your ceiling.

So, the model treats your labels as truth, learns your mistakes as rules, and a bigger model only learns them better. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is biased data? Let's try to understand. In a simple way. Here is a hiring dataset.

Ten years of real decisions. No typos, no wrong labels. Every row is accurate. And the model you train on it quietly rejects women.

Because training has one job. Find whatever predicts the label. Past managers mostly hired men. So the word man predicts the word hired.

The model copies that. It never sees reasons. It sees a column and an outcome. To the math, prejudice and skill look exactly the same.

So delete the gender column. Train again. The model comes back just as skewed. Because other columns carry it.

A women's college. A two year gap. A hobby, a zip code. The model rebuilds what you deleted from what you kept.

Those are proxies. And your score looks fine. Ninety two percent. But that is an average, and the group is small, so its errors barely move it.

So split the test set by group and score each one. Ninety six percent here. Seventy one there. The gap was always there.

You just never asked. And it hardens. The model rejects them, so they never get hired, so no success records appear. Next year's data agrees.

It manufactures its own proof. There is no fair button. Equal error rates, or equal offer rates? They conflict.

You pick one. That is a human decision, not a loss function. So, accurate data can still be unfair, proxies survive deletion, and only a per group score will show it. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is a representative dataset? Let's try to understand. In a simple way. You can never collect the world.

You collect a sample. The model treats that sample as the whole world, because it never sees anything else. Where you collected decides who got in. Photos from one sunny city.

Reviews from angry customers. The channel is a filter you never noticed. Nineteen thirty six. A magazine mails ten million ballots.

Two million come back. It predicts Landon. Roosevelt wins forty six states out of forty eight. Their addresses came from car registrations and phone books.

In the Depression, that is whoever could afford a car. Wealthy sample. Poor country. So two million replies bought nothing.

Size kills noise. It does nothing to skew. Same channel, more rows, more confident and still wrong. And you cannot spot this from inside.

Rows never collected leave no empty cell. Your data looks complete, because what is missing is simply absent. Bring in an outside number. Split by age, compare your shares to census figures.

Yours says a third are teenagers. The world says a tenth. You can reweight. Count the thin group's rows for more, until the shares match.

But weights only stretch what you have. Zero rows stay zero. Even a perfect sample goes stale. You collected in winter.

Now it is summer and prices moved. The world drifted off your snapshot. That is distribution shift. So, your data is a sample, the model believes it is the world, and only an outside yardstick shows the difference.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a held out test set? Let's try to understand. In a simple way. You train the model on all of your data.

Now you want to know how good it is. So you grade it. On the same data. That score is a lie.

A model that just memorized every row would ace it. And still be useless on anything new. So before any training starts, you cut a slice off your data and lock it away. Usually around twenty percent.

The model never sees it. You train on what is left. On grading day, you run the locked slice through. Those rows are new to the model, so that score estimates real life.

But you also need a slice to make choices with. Which model, how many layers. So the split is three ways. Train, validation, and test.

You tune on validation as often as you like. The test set stays sealed. You open it once, at the very end. Why once?

Because every peek leaks. You read the test score, you change something, and your choice now carries that slice inside it. The other break is leakage. The same row, or a near copy, sitting on both sides of the split.

The model recognizes it, and the score inflates. So split by the thing that repeats. Every scan of one patient goes to one side. And if you predict the future, split by time, not at random.

So, hide a slice, train on the rest, open it once. It is the only honest score you will get. Quick check now. One question is coming up.

Let's see if it clicked.