Voice agents, end to end

How does digital audio work? Let's try to understand. In a simple way. Say a word out loud.

Your vocal cords shove the air, and that shove travels outward as a wave. Squeeze, stretch, squeeze. Sound is just air pressure, wobbling. Inside every microphone sits a thin sheet called a diaphragm.

The passing wave pushes it back and forth, and that motion becomes a wobbling voltage. Still not numbers. So the machine measures that voltage. Then measures it again.

Over and over, on a strict clock. Each reading is one number, called a sample. How often? For speech, usually sixteen thousand times every second.

That is called the sample rate. One second of talking becomes sixteen thousand measurements. Why that many? To catch a wobble, you must measure at least twice as fast as it wobbles.

Speech runs to about eight thousand wobbles a second. Double it, and you get sixteen thousand. Every reading also gets rounded to the nearest rung on a ladder. Sixteen bits gives sixty five thousand rungs.

Fine enough that the rounding error sounds like faint hiss. And these numbers never arrive one by one. They pile up, then get handed over in small chunks, often twenty milliseconds each. Three hundred twenty samples per delivery.

Here is the honest picture. No words. No letters. No meaning.

Just a river of numbers. Every clever thing a voice agent does is built on that river. So, hearing is measuring. Air nudges a sheet, a clock samples the wobble, and your voice arrives as numbers.

Quick check now. One question is coming up. Let's see if it clicked.

How does text to speech work? Let's try to understand. In a simple way. Hearing was measuring.

A microphone turned air into numbers. Speaking runs that trip backwards. The machine must invent the numbers, tens of thousands of them for every second it talks. The old trick was recording a person saying thousands of snippets, then gluing them together.

That is concatenative synthesis. It works, until you hear the seams. But nobody can record every sentence. Your name, a new address, a number nobody has said yet.

So the machine must generate speech it never recorded. Modern synthesis splits the work in two. First, decide how the line should sound. Then turn that plan into audio numbers.

Two models, handing over in the middle. The words never say how to say them. Rising at the end makes a question. Falling makes a statement.

Pitch, pace, pauses, all guessed. That guessing is called prosody. Then the second model paints the waveform, sample by sample. It is called the vocoder, and it is the only piece that hands out real audio numbers.

And whose voice comes out? A short list of numbers, taken from a few seconds of someone talking. Swap the list, and the model speaks as someone else. The last step is physics.

The numbers become a wobbling voltage, the voltage shoves a paper cone, and the cone shoves the air. The microphone, in reverse. So, speaking is generating. Decide how it should sound, paint the numbers, push the air.

Nothing was ever recorded. Quick check now. One question is coming up. Let's see if it clicked.

How does the simple ear to mouth loop work? Let's try to understand. In a simple way. You have two halves already.

Ears that turn air into numbers. A mouth that turns numbers back into air. But that is not a conversation. Something must sit in between.

The simplest voice agent is three boxes in a row. Speech to text turns your numbers into words. A language model reads them and writes a reply. Text to speech turns that reply back into sound.

So notice what an agent really is. Not one clever model that hears and speaks. Three separate models, standing in a line. And look at what travels between them.

Not audio. Text. Plain words. Text is the only thing these boxes hand each other.

Which means the first box throws things away. You shouted. You hesitated. You were joking.

None of that survives. The model just reads the word, yes. Here is the part that makes it simple. Each box waits for a complete thing.

Speech to text waits until you stop talking. The model waits for the finished sentence. So the loop turns one way at a time. It listens, or it talks.

Never both. Like a walkie talkie. You finish, it answers, you answer back. And because nothing overlaps, the waits stack.

Hearing, then thinking, then speaking. Every box adds its own delay to the total. So, the simple loop is a relay. Three models, text between them, each waiting its turn.

Ear, to mouth, and round again. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is conversational lag? Let's try to understand. In a simple way. When two people talk, the gap between one stopping and the next starting is about two hundred milliseconds.

A fifth of a second. Sometimes it is zero. Sometimes they overlap. That same gap shows up in every language anyone has measured.

You have heard it your whole life, so you never think about it. But you expect it. So when the gap stretches, your brain does not think, computation. It reads the silence as a message.

He is hesitating. She disagrees. Maybe they did not hear me. Half a second, and the reply still feels like a reply.

One second, it feels like thinking. Past two, it feels broken. Here is the cruel part. The listener's clock starts the moment you stop talking.

Not when the machine notices. Everything it does after that is already inside the gap. And the machine has real work to do in there. Hear you.

Think. Start speaking. All of it lands inside that same fifth of a second. Run long, and people repeat themselves.

Hello? Are you there? That new speech arrives just as the answer starts. Now you are both talking.

Which is why lag is not a speed problem. It is a social one. Silence is all the listener has, and empty silence reads as something being wrong. So, lag feels awkward because you already had a number in your head.

Two hundred milliseconds. Everything past it, your brain reads as meaning. Quick check now. One question is coming up.

Let's see if it clicked.

How does realtime transcription work? Let's try to understand. In a simple way. The simple version just waits.

You talk, the chunks pile up, and nothing comes out. Only when you stop does it hand back a sentence. Why wait? Because the model reads the whole recording at once.

To decide one word, it looks at the audio on both sides. And the future really does carry information. Ice cream, and I scream, are the same sound. That instant cannot tell them apart.

Only what follows can. Realtime transcription flips it. Chunks go in as they arrive, twenty milliseconds at a time, and words come out while you are still talking. But something is missing.

The audio after this moment has not happened yet. So the model can only look backwards. The right side is gone. So give it a small peek.

Hold each word back a couple hundred milliseconds, and it hears a little of what follows. That is the lookahead window. Wider peek, better words, more delay. That is the trade, and it is a dial you set.

It also changes its mind. Early words are partials. Drafts. I scream, becomes, ice cream, once more audio lands.

A word goes final when nothing can move it. Here is the payoff. By the time you stop talking, nearly the whole sentence is already transcribed and gone. Only the tail is left.

So, realtime transcription is the same job, without the future. Peek a little, guess early, fix it later. The words leave before you do. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is word by word generation? Let's try to understand. In a simple way. Your words are now text.

The model reads them and must reply. It does not compose the whole answer at once, but builds it piece by piece. Each piece is a token, a word or a chunk. Looking at everything so far, the model produces just the next token, then stops.

Then the clever part. It glues that token onto the end and feeds it all back in. Now it guesses the next, given what it wrote. How does it choose?

It scores every piece in its vocabulary, tens of thousands, then picks from the top of the list. High score, likely word. It can always grab the highest score, safe but stiff. Or roll a weighted dice among the top few.

That pinch of randomness varies the wording each time. The loop keeps running. Token, append, guess again. It halts only when it produces a special stop token, its way of saying my turn is done.

Why not write it all at once? Because each word leans on the ones before it. Word five cannot exist before word four. So the pieces arrive in order.

Here is the payoff for a voice agent. You do not wait for the whole answer. The tokens spill out in a steady trickle, ready as they land. So the reply is built token by token.

Guess one, append it, feed it back, repeat until the stop token. A sentence, left to right, live. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is time to first sound? Let's try to understand. In a simple way. The language model is writing its reply, one token at a time.

But a token is just text. To speak out loud, you need a voice to turn text into sound. The simple way waits for the whole reply. Every word arrives, the sentence finishes, and only then does the voice begin turning it into sound.

But writing a full sentence takes time. So the person who just spoke hears nothing. A long, silent gap before the first sound. The fix is to stop waiting.

The tokens already spill out in a trickle. So feed them to the voice the moment they land. But the voice cannot speak half a word. So it holds the tokens until a natural break, a comma or a short clause, then turns that chunk into sound.

That first chunk is tiny, so it becomes audio almost at once. The speaker starts talking on it while the rest of the reply is still being written. How big is each chunk? That is a dial.

Too small, and the voice sounds choppy, its rhythm broken. Too big, and you wait longer for the first sound. Now three things run at once. The model writing, the voice synthesizing, the speaker playing.

Each stage hands its chunk down the line. So, time to first sound is the gap from the first token to the first bit of audio. Stream the tokens, chunk them at natural breaks, and that gap shrinks. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is the polite interruption problem? Let's try to understand. In a simple way. Picture the agent in the middle of a long answer.

Halfway through, you jump in with a new question. What should it do? A naive machine just keeps going. It finishes its whole sentence, right over you.

It feels rude. And worse, it never heard your question. Real people do the opposite. The moment someone cuts in, you stop, mid word if you must, and hand over the floor.

That is the reflex we want. So the agent keeps its ear open the whole time it speaks. The instant it hears you begin, it interrupts itself. That is barge in.

But there is a trap. The voice was generated ahead and sits in a queue, waiting to play. Stopping the generator does nothing. The queued speech keeps pouring out.

So it throws the queue away. Dump every buffered chunk, kill the playback, fall silent inside a fraction of a second. Only then does it feel instant. It also drops the rest of the answer it was about to give.

That turn is over. Your new words become the next thing it hears. One catch. Not every sound is an interruption.

A cough, or an uh huh, just means you are still listening. Stop on those and it gets twitchy. It waits for real speech. So, the polite interruption problem is this.

Listen even while you talk. Stop the instant you are cut off. And clear the queue, so silence lands right away. Quick check now.

One question is coming up. Let's see if it clicked.

How does listening while already speaking work? Let's try to understand. In a simple way. To catch you the instant you cut in, the agent leaves its mic open while it speaks.

But that open ear brings a problem. Its own voice leaves the speaker, crosses the room, and loops back into the open mic. So the loudest thing it hears is itself. That returning sound looks just like a person talking.

The agent could think you barged in and stop, hearing only itself. The easy fix is to mute the mic while it talks. But then it is deaf, and a real interruption sails right past. Here is the key.

The agent knows exactly what it is playing. It keeps a perfect copy of that outgoing sound: the reference. So it lines the reference up against the mic and subtracts it. Whatever is left must be you.

This is acoustic echo cancellation. But the room bends the sound: a small delay and echoes off the walls. An adaptive filter learns that delay and echo, and reshapes the reference to match. When you both talk at once, the mic holds two voices mixed.

The filter must freeze its learning, or it warps and erases you as well. Even then, a faint tail of echo slips through. A final suppressor mops it up, so the words sent on are cleanly yours. So, listening while speaking means canceling your own voice from the mic: keep a copy, learn how the room bends it, and only you are left.

Quick check now. One question is coming up. Let's see if it clicked.

How does a realtime voice pipeline work? Let's try to understand. In a simple way. A voice agent is a chain of stages.

Your voice goes in one end, and its reply comes out the other. Let's follow the whole trip. First an ear turns your speech into text. A language model reads it and writes a reply.

A voice turns that reply back into sound. The lazy way is a relay. Each stage waits for the one before it to completely finish before it starts. Ear, then brain, then voice.

But those waits add up. A pause to hear, a pause to think, a pause to speak. Stacked together, that is seconds of dead silence. So real pipelines never wait for done.

Each stage passes on its work the moment it has a scrap, not when it is finished. That means the stages run at the same time. Like an assembly line, every station stays busy at once. This overlap is called pipelining.

So the delay you feel is just the first scrap reaching the end. Not the sum of every stage. That is the whole trick. And it all runs in a loop.

The reply plays while the ear is already open again, ready for your next turn. One more piece ties it together. A conductor watches for when you stop talking, then tells the chain to start. Timing is everything.

So the full pipeline is one streaming loop. Ear, brain, voice, all overlapping, so the first sound comes back fast. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a phoneme? Let's try to understand. In a simple way. A machine cannot simply read letters aloud.

English spelling lies. Though, through, and tough share four letters, yet each ends in a different sound. So the machine works in sounds instead. A phoneme is the smallest unit of sound in speech, not a letter on the page.

English has about forty. Every word is a short string of them. Sea and see are spelled differently, yet you say the very same sounds. Spelling shifts; the phonemes stay put.

So the synthesizer's first job is to turn letters into phonemes. Letters on the page are graphemes. This step is grapheme to phoneme. How?

First it looks in a dictionary of pronunciations, a lexicon. Cat maps to three sounds. Common words are checked by hand. But no dictionary holds every word.

A new brand, a strange surname? A small model guesses the sounds straight from the spelling. One spelling can hide two sounds. Read a book, or read it yesterday.

Lead the team, or a pipe of lead. Only the sentence decides. Sounds are not the whole story. Stress matters too.

You keep a record, but you record a song. Same spelling, the stress moves. Now the line is a clean row of sounds, each marked with its stress. That row is what the next stage turns into audio.

So, a phoneme is a unit of sound, not a letter. Turn spelling into sounds, let context settle the tricky words, and the voice can finally speak. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is a spectrogram? Let's try to understand. In a simple way. The sound wave is a firehose.

Tens of thousands of numbers a second. Predicting that directly is brutal, so the model paints a picture of the sound instead. That picture is a spectrogram. Time runs along the bottom, pitch climbs up the side, and brightness shows how loud each pitch is at each moment.

How? Chop the wave into short overlapping windows. For each, the Fourier transform asks which pitches are inside and how loud. Stack those columns and the picture appears.

One twist. Your ear does not hear pitch evenly. Low notes spread out, high notes bunch up. Squeeze the pitch axis to match, and you get a mel spectrogram.

Why a picture, not the wave? It is far smaller. One thin column every ten milliseconds, about eighty rows. Compact, yet it still holds the voice.

So the first model's real job is this. Take the phonemes and the prosody, and paint the mel spectrogram. That painter is the acoustic model. But the picture is not the sound.

It keeps how loud each pitch is, yet throws away the exact wiggle, the phase. It is a recipe, not the sound. That is why the vocoder comes next. It reads the picture and invents the missing wiggle.

Only then do you get real audio numbers. So, a spectrogram is a picture of sound. Time across, pitch up, brightness for loudness. The model paints it, the vocoder turns it back into a wave.

Quick check now. One question is coming up. Let's see if it clicked.

How does a latency budget work? Let's try to understand. In a simple way. A voice reply has to start almost instantly.

Take longer than about eight hundred milliseconds, and it feels slow. That is your ceiling. Think of those milliseconds like money. You get a fixed amount to spend, and every stage on the path charges some.

So where does the time go? The wait until you clearly stopped talking. Speech turned to text. The model's first word.

That word turned to sound. Each one has a price tag. The end wait, two hundred milliseconds. Transcription, one hundred.

The first word, three hundred. The first sound, one hundred fifty. Now add every price along the path to the first sound. Two hundred, plus one hundred, plus three hundred, plus one hundred fifty.

Seven hundred fifty. Seven fifty fits under the ceiling, but only just. One heavier stage, and you spill over. One catch.

You only count the time up to the first sound. The rest of the reply plays out loud while the clock has already stopped. So find where the milliseconds pile up. Here the model's first word is the fattest line, three hundred on its own.

Trim the biggest spender first. Swap in a smaller model, or start on a half heard sentence, and you win time back. The sum now ducks well under the ceiling. So a latency budget is simple.

Add each stage's time to the first sound, keep the total under your ceiling, cut the biggest cost when it spills. Quick check now. One question is coming up. Let's see if it clicked.

How does filler audio like umm work? Let's try to understand. In a simple way. When you stop talking, the real answer is not ready.

The machine still has to understand, think, and speak. That leaves a silent gap. People fill it without thinking. You say umm, or well, keeping the floor while your brain catches up.

A voice agent copies this. The moment you finish, it plays a tiny filler at zero delay, while the pipeline works behind it. How is it instant? The filler is not made on the spot.

It is a short clip, recorded ahead and cached. The clip carries no answer. Its only job is to cover the gap and hold the floor, so the pause feels like thinking. The real reply is being built.

When ready, it is stitched on right after the filler, so it sounds like one voice. One trap is repetition. The same umm every time sounds broken. So the agent keeps a small bag of fillers and rotates them.

Better still, match the filler to the task. Say, let me check that, before a slow lookup. Say, sure, for a quick yes. But there is a catch.

Filler buys time, it adds nothing. If the answer was already fast, the filler just delays it. So, filler audio is a canned sound that plays at once, holding the floor while the real reply is built. Skip it when the answer is already quick.

Quick check now. One question is coming up. Let's see if it clicked.

How does turn taking work? Let's try to understand. In a simple way. Before it can answer, the agent must catch the moment you finish.

Start too early, it cuts you off. Too late, it feels dead. This job has a name. Endpointing.

Also called turn detection. Its question is simple. Has your turn ended? The first tool is voice activity detection.

A fast check tagging each slice of sound as speech, or silence. It hears energy, not words. So the naive rule is simple. The moment the microphone goes quiet, call the turn finished, and start replying.

But silence does not mean finished. You pause to think in the middle of a sentence. Fire on that gap, and the agent talks over you. The fix is a silence timer.

Do not react to the first pause. Wait a set stretch of quiet, maybe seven hundred milliseconds, before calling it done. That wait is one dial with two costs. Set it short, the agent is snappy, but it interrupts.

Set it long, it waits politely, but feels slow. The smarter fix listens to the words. A turn detection model reads your sentence so far. Book a table for sounds unfinished.

For four people sounds complete. It also reads your pitch. A rising tone means more is coming. A falling tone means you are done.

The model hears the melody, not the transcript. So, put together. The detector spots the silence, a timer waits, and a model weighs your words and tone. Only then does the agent speak.

Quick check now. One question is coming up. Let's see if it clicked.