Whisper, dissected

What exactly is a Mel spectrogram? Let's try to understand. In a simple way. Sound is pressure.

A microphone feels the air push and pull and writes one number per moment. Plotted, it is a wiggly line. The paper resamples every recording to sixteen thousand samples per second. One second of speech becomes a list of sixteen thousand pressure readings.

But that list hides what matters. Speech blends slow and fast vibrations. We want the frequencies present, and their strength, moment by moment. So take a slice just twenty five milliseconds long and measure the strength of each frequency inside it.

That gives one thin column. Then slide the window forward ten milliseconds and repeat. Windows overlap, and a fresh column lands every ten milliseconds: a hundred columns per second. Each column holds eighty numbers, one per band.

The bands follow the Mel scale: low frequencies spread out, high ones squeeze together, much like human pitch perception. Each strength is stored on a log scale, so quiet sounds stay visible beside loud ones instead of being crushed flat. Now count one second of speech. Sixteen thousand raw readings go in.

Out comes a grid of a hundred columns by eighty bands: eight thousand cells. Finally the whole grid is rescaled to sit between minus one and one, roughly centered on zero, a small steady range that networks digest well. Speech is now a picture. Time runs left to right, pitch bottom to top, brightness is loudness.

That grid is what everything later reads. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is sequence to sequence speech recognition? Let's try to understand. In a simple way. Last time, sound became a picture: eighty frequency bands, one column every ten milliseconds.

Now the question is what a network has to produce from that grid. Count the two sides. Thirty seconds of audio is three thousand columns. The words spoken might be seventy tokens, the little pieces text is chopped into.

Thousands in, dozens out. And nothing in the data says which columns belong to which word. The transcript is just text. No frame is labelled.

The model must discover the pairing itself. This shape has a name: sequence to sequence. One sequence in, a shorter one out, no fixed rule linking positions. The score is word error rate.

Count the substitutions, deletions and insertions that turn the output into the reference, divided by reference word count. Lower is better. Ten reference words. The system swaps one, drops one, adds one.

Three edits over ten words: thirty percent word error rate. Yes, it can exceed one hundred. Here is the trap. Every difference counts, even formatting.

Write twenty five as digits while the reference spells it out, and that is two errors for a perfectly heard sentence. So the authors built a text normalizer that strips style quirks from both sides before scoring. On some datasets, that alone cut measured error by up to half. So remember: word error rate mixes hearing mistakes with writing conventions.

Next lesson, the two part network that actually turns frames into tokens. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the encoder-decoder Transformer? Let's try to understand. In a simple way. Last lesson: thousands of frames in, a few dozen tokens out.

That mismatch is why the network has two parts. One reads, one writes. The encoder reads the whole spectrogram at once and turns each frame into a vector: a list of numbers summarising that slice of sound in context. Context comes from attention.

Every frame scores every other frame for relevance and blends in the useful ones. So one frame's vector carries hints from its neighbours. Concretely, thirty seconds of audio is three thousand columns. Out comes a long row of vectors, one for every two columns.

Still no text. Just sound. The decoder writes the transcript one token at a time, seeing the tokens so far and guessing the next. That is exactly what a language model does.

A plain language model only guesses likely words. So the decoder also attends across to the encoder's vectors, scoring each one for relevance at every step. That is cross attention. Tokens so far: the cat sat on the.

Audio vectors near second four light up for the sound mat. So it writes mat, not a merely likely word like floor. So the decoder is a language model conditioned on audio. Grammar from the text it has written, grounding from the sound it can hear.

Both at every step. Radford and colleagues chose this design because it was known to scale reliably, not because it was new. What they built on top comes next. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is weak supervision? Let's try to understand. In a simple way. Last lesson's encoder-decoder learns from pairs: audio in, matching text out.

Where do those pairs come from? The paper sorts the answers into three kinds. First, supervised: recordings with transcripts humans wrote or checked. SpeechStew pooled seven such datasets, about five thousand hours in total.

Clean, trustworthy, and small, because careful transcription is expensive. Second, unsupervised: audio with no text at all. Recent systems trained on a million hours of unlabeled speech, learning rich internal representations of how speech sounds. But something is missing.

Without transcripts, nothing teaches the model to write. A strong encoder emerges, but no equally good decoder. A million hours, and not one word out. So in practice you fine-tune: train that pretrained model further on one small supervised dataset, until a working decoder appears.

Here is the trap. Fine-tuning teaches the quirks of that one dataset along with the task. The model shines on matching test sets and stumbles elsewhere. Vision gives the proof.

Fine-tuning an image model on ImageNet lifted accuracy there by nine point two percent, yet averaged across seven other datasets, the same objects, it gained nothing. Now the third kind. The internet holds audio that already comes with some transcript attached. Nobody verified it.

Some is machine made, some plain wrong. That is weak supervision. So the paper bets: accept weaker labels in exchange for far more of them, and skip fine-tuning entirely. One model, used everywhere.

Next lesson: making that pile usable. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is Whisper's training data? Let's try to understand. In a simple way. Last lesson ended on a trade: accept messy internet transcripts, get scale.

The result: six hundred eighty thousand hours of audio paired with text. But much of that text was written by other speech recognizers, not people. Train on their output and you learn their habits, not how people actually write. Machine transcripts give themselves away.

They drop what audio cannot predict, like punctuation and capitals. So transcripts in all capitals, or all lowercase, were thrown out. Next, an audio language detector listens to the recording while a text detector reads the transcript. If spoken and written languages disagree, the pair leaves the recognition data.

One exception. Foreign speech with an English transcript is a ready made translation example: audio in one language, English text out. Those pairs were kept. Then deduplication, but fuzzy.

Internet transcripts get copied with small edits, so exact matching misses repeats. Fuzzy matching catches near copies too, and those were removed. The survivors were cut into thirty second segments, each paired with the transcript spoken inside that window. Silent segments were sometimes kept too, as examples of no speech.

One filter needed a model first. Train an initial version, measure its error rate on each source, inspect the worst by hand, and remove the low quality ones. The tally: one hundred seventeen thousand hours cover ninety six languages beyond English, one hundred twenty five thousand hours pair foreign speech with English text, and the large majority is English. Quick check now.

One question is coming up. Let's see if it clicked.

How does Whisper's multitask format work? Let's try to understand. In a simple way. Last time we saw the six hundred eighty thousand hours.

How does one model do many jobs? A traditional stack chains separate components: one detects speech, one identifies the language, one transcribes, others add translation and timing. Whisper's answer is almost cheeky. The decoder already writes tokens one at a time.

So describe the entire job as tokens too, all inside a single sequence. The sequence opens with a start token. Then a language token names what is being spoken. There are ninety nine of these, one for every training language.

If the thirty second clip contains no speech at all, a special no-speech token goes there instead. Predicting that token is voice detection, handled for free. Next comes a task token: transcribe, or translate. Translate always means the same thing here, foreign audio in, English text out.

Then, timing. A no-timestamps token asks for plain text. Otherwise, time tokens are woven between the words, each one rounded to the nearest twenty milliseconds. Put it together.

French speech, English text wanted, no timing. The decoder writes: start, the French token, translate, no-timestamps, then the English words, then an end token. One more ingredient. Sometimes the transcript of the previous segment is added to the decoder's context.

The model learns to lean on what was just said. Step back. Voice detection, language identification, transcription, translation, timing. Five separate components in the old stack, now just one string of tokens read by one model.

Quick check now. One question is coming up. Let's see if it clicked.

How does Whisper's training recipe work? Let's try to understand. In a simple way. Dataset done, token format done.

Now the training itself. The surprise is how little there is. Radford and colleagues trained five sizes of one design: Tiny, Base, Small, Medium, Large. The smallest holds thirty nine million parameters; the largest about one point five billion, forty times more.

The recipe is textbook. AdamW, a standard rule for nudging weights. The learning rate warms up for two thousand forty eight updates, then decays in a straight line to zero. Each batch holds two hundred fifty six thirty second segments, so one update hears over two hours of audio.

After about a million updates, the model has seen the dataset just two to three times. And the missing part: no augmentation, no artificial noise added to the audio; no regularization, no extra brakes on memorizing. The bet is that six hundred eighty thousand hours carries enough variety on its own. One glitch slipped through.

Early models invented plausible speaker names nobody said. Internet transcripts often label who is talking, so the model learned to guess names it could not hear. The fix was surgical. Fine-tune briefly on only the transcripts without speaker labels.

The habit disappears. One exception came later. Large V two trained two and a half times longer, and for that run the tricks returned: masked spectrogram patches, randomly skipped layers, varied tokenization. So the recipe is boring on purpose.

The claim rides entirely on the data. Next, whether that bet paid off. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is effective robustness? Let's try to understand. In a simple way. Whisper is trained.

Time to score it. The paper sets one strict rule, called zero-shot: never touch any benchmark's training data. Every test is a cold open. Its rival is wav2vec two point zero, pretrained on unlabeled audio as lesson four described, then fine-tuned on LibriSpeech: a classic benchmark of clean audiobook speech.

Home turf. On LibriSpeech clean speech, they tie. Both score exactly two point seven percent word error rate. Judged by this one number, the two models are identical.

Now leave home. The paper runs both models on thirteen other test sets: phone calls, meetings, accented speech, noisy rooms. Same task, and no retraining allowed for anyone. The tie shatters.

On the Common Voice dataset, wav2vec's error rate is twenty nine point nine percent. Whisper's is nine point zero. Averaged across all thirteen, wav2vec sits at twenty nine point three, Whisper at twelve point eight. Fifty five point two percent fewer errors, from the same home score.

Why? Fine-tuning taught wav2vec the quirks of one dataset, not speech in general. Whisper's two point seven was earned with no LibriSpeech help at all, so it travels. Effective robustness makes this precise: how much better a model does off home turf than its home score predicts.

Plot LibriSpeech score against the average everywhere else. Humans trace a line on that plot: slightly worse away from clean audio, never a collapse. LibriSpeech-trained models float far above it. Zero-shot Whisper lands close to the human line.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is Whisper's multilingual scorecard? Let's try to understand. In a simple way. So far, every score has been English.

But one hundred seventeen thousand hours of Whisper's data cover ninety six other languages. Time to grade them. First, recognition. On Multilingual LibriSpeech, read audiobooks in many languages, zero-shot Whisper scores seven point three word error rate, beating the supervised systems.

On VoxPopuli, European parliament speech, it scores thirteen point six. Rivals trained on that exact data get around eight. Home advantage still wins. Why the spread?

Mostly hours. Word error rate halves for every sixteen times more training data in a language. Hours predict most of the score. A few do worse than their hours predict: Hebrew, Telugu, Chinese, Korean.

Unusual scripts, far from English, and a tokenizer, the tool chopping text into units, that matches their scripts poorly. Now for translation. Foreign audio in, English text out, using the translate token from before, same single model. On the CoVoST two benchmark, Whisper hits twenty nine point one BLEU, a translation score where higher is better.

A new zero-shot state of the art. But the win is lopsided. On low-resource languages, those with little data, Whisper leads by six point seven BLEU. On high-resource languages it actually trails.

One outlier exposes the data itself. Welsh supposedly has nine thousand translation hours yet scores only thirteen BLEU. Most of that audio was really English, mislabeled as Welsh. The honest ledger: strong where data is scarce, beaten on rivals' home turf, hours rule the rest.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is real-world robustness? Let's try to understand. In a simple way. Every score so far came from tidy benchmark audio.

Real recordings have chatter behind the speaker and run for an hour. Does Whisper survive that? First, noise. The authors mixed pub noise into clean LibriSpeech audio, louder and louder.

Signal to noise ratio measures how much louder the speech is than the chatter. On nearly clean audio, several LibriSpeech trained models beat Whisper. But once the speech is under ten decibels above the noise, every one falls behind it. Second, length.

Whisper reads only thirty seconds at a time, the size of its training segments. How do you transcribe an hour long interview? By buffering. Transcribe one window, read the timestamps the model itself wrote, slide forward to where the text ended, then repeat.

Left alone, this drifts, so guardrails were added. Keep five candidate transcripts, not one. When confidence drops, or the output repeats until it compresses suspiciously well, retry with extra randomness. The payoff is modest and honest.

Across seven long form datasets, average word error fell from eleven to ten. Some tricks hurt on individual datasets; they win only on average. Now the humans. Twenty five recordings from the Kincaid forty six set went to five professional transcription services.

The computer assisted one beat Whisper by just over one point. The purely human ones were a fraction of a point better. The quiet headline: on real recordings, Whisper sits within touching distance of professional humans. Next, what scale actually bought.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is an ablation? Let's try to understand. In a simple way. Whisper's results are strong, but which ingredient did the work?

An ablation answers that: change one thing at a time. This paper turns three dials. Dial one is model size: Tiny at thirty nine million parameters up to Large at about one and a half billion. Bigger keeps helping on every task except one.

English flattens, and the explanation is saturation: the model is nearing human level, so little error is left. Dial two is data. Medium sized models train on slices, from about three thousand four hundred hours to the full six hundred eighty one thousand. English word error rate falls from thirty point five to ten point nine at fifty four thousand hours.

The full dataset, over twelve times more, buys one more point: nine point nine. Away from English, data rules. Multilingual error falls from ninety two point four to twenty nine point two; translation climbs from near zero to twenty four point eight BLEU. Dial three: does cramming ninety six languages and translation into one model hurt English?

They match compute first, since joint models spend only sixty five percent of training on English. Small joint models do lose: the extra tasks crowd English out, negative transfer. But at the largest scale the sign flips, and joint beats English-only. So scaling bought robustness on every task, plus the multitask bet itself, which only pays at the top.

The one thing it cannot buy: progress past human level. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is Whisper's fine print? Let's try to understand. In a simple way. Eleven lessons of results.

Now the authors list what still breaks. That honesty is how the paper closes, and how we will. First, decoding. The model still falls into repeat loops, drops first or last words, or hallucinates text unrelated to the audio.

The heuristics reduced this, not removed it. Second, the data is heavily English: most languages have under one thousand hours. Error halves with sixteen times more data. Those languages never got theirs.

Third, naming the spoken language is weak: sixty four point five percent accuracy, partly because twenty of the benchmark's languages never appeared in training. Two open questions. Fine-tuning with good labels would likely help; it went unstudied. And nobody knows if the gains live in the encoder, the decoder, or both.

Others answered fast. WhisperX first finds where speech actually occurs and cuts the audio there, so segments run in parallel batches, and aligns speech sounds for word level timestamps. Twelve times faster. Distil-Whisper let Whisper label huge piles of audio, kept only outputs passing an error check, and trained a smaller student.

About half the parameters, nearly six times faster, within one percent on unseen data. Notice the pattern. The paper's admitted weaknesses became the follow-ups' abstracts. The distilled student even hallucinates less on long audio than its teacher.

That is the whole story. No new architecture, no clever objective. Six hundred eighty thousand messy hours, one token format, honest zero-shot evaluation. That was enough.

Quick check now. One question is coming up. Let's see if it clicked.