Context engineering

What exactly is the context window? Let's try to understand. In a simple way. Here is a strange fact about a language model.

On its own, it remembers nothing. Every time you send a message, it wakes up with a blank mind. So how does it recall what you said earlier? It does not.

Instead, the whole conversation so far is quietly sent again, packed into every new request. That bundle of text you hand the model has a name. The context window. It is simply everything the model can see at this moment.

Inside it sits your latest question, the earlier back and forth, and the reply the model is writing. All of it, in one place. But this window is not endless. It has a fixed size, measured in tokens.

Maybe a hundred thousand of them. Like a page that can only hold so much. And here is the key idea. This window is the model's entire world.

If a fact is not inside it, the model simply does not know that fact exists. There is no saved memory tucked away somewhere. On every single turn, the model reads the whole window again, from the first word to the last. So what happens when a long chat grows past that limit?

The oldest lines fall out of the window. And once they are gone, the model has forgotten them. So the context window is the model's short term memory. A fixed space, re-read fresh every turn.

Whatever fits, the model knows. Whatever falls out, it forgets. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is one token? Let's try to understand. In a simple way. Before a model reads your text, it chops it into pieces.

Each piece is one token. So a token is just a small chunk of text, sitting in the model's list. The model keeps a fixed list of pieces, its vocabulary. A common word like cat is on that list all by itself.

So cat is one single token. Here is a small surprise. The blank space before a word usually joins that word. So a space plus cat counts as one token, not two.

A long or unusual word is not on the list as a whole. So it breaks into several tokens. The word unbelievable splits into three pieces. One word, but three tokens.

Punctuation counts as well. A comma, a period, a question mark, each one is its own token. Even an emoji is one or more tokens. Numbers and code are messier.

A long number can break into several tokens, sometimes digit by digit. Rare symbols and code split hard too. So how do you count them fast? A handy rule for English.

One token is about four characters, or roughly three quarters of a word. A hundred words is near one hundred and thirty tokens. So one token is one piece from the model's fixed list. Common words are one token, rare ones are many, and spaces and marks count too.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a token budget? Let's try to understand. In a simple way. Every model has a context window.

It is the space where your whole prompt lives. And that space is fixed. It holds only so many tokens, no more. Now here is the catch.

Many things must fit in that one space. Your instructions, some examples, retrieved documents, the chat history, the user's question. All share one budget. And there is one more guest.

The answer needs room too. The model writes its reply into the same box. Fill it to the brim, and nowhere is left to respond. So every token you add has a price.

A long instruction eats the same space as a short one. Filler crowds out something useful. Spending wisely means asking one question of each piece. Does this help answer the task?

If yes, it stays. If not, it goes. Aim for signal, not size. Ten crisp lines can beat two pages of the same idea.

Pack the most useful meaning into the fewest tokens. That is density. Piling on more is not free either. A window stuffed with weakly related text can bury the one line that mattered.

The model's attention spreads thin. There is a second bill. More tokens cost more money per call, and they take longer to process. A lean prompt is cheaper and faster.

So, the window is a fixed budget everyone shares. Keep what earns its place, pack meaning tightly, and always leave room for the answer. Quick check now. One question is coming up.

Let's see if it clicked.

How does vector search find relevant facts? Let's try to understand. In a simple way. The old way was keyword search.

It looks for the exact words you typed. So a question about a flat bike tire misses a note on repairing a punctured wheel. Same idea, no shared words. Vector search fixes this by storing meaning as a location.

Every fact becomes a point in a huge space of numbers. What the fact means decides where it lands. And here is the trick. Points that mean similar things end up close together.

So all your notes about bike repair gather in the same little neighborhood. Now you ask a question. That question is turned into a point too, dropped into the very same space. To answer it, the system looks around that point.

It grabs the stored facts sitting nearest to your question. Those are the relevant ones. But how do we measure near? Each point is really an arrow from the center.

We compare the angle between two arrows. A small angle means close meaning. That score is called cosine similarity. It does not stop at one.

It returns the top few closest facts, say the nearest three. Enough to answer, nothing more. Because everything is matched by meaning, the words no longer have to line up. Flat tire quietly finds punctured wheel.

That is the whole point. So vector search turns text into points, places similar meanings side by side, and pulls the nearest ones to your question. Search by meaning, not by matching words. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is chunking? Let's try to understand. In a simple way. Your AI needs to answer from your own documents.

But those documents are huge. Manuals, reports, whole books. Far too big to hand the model at once. The context window is fixed.

A fifty page manual would never fit. And even if it did, the model would drown in text it does not need. So before anything, we chop each document into small pieces. These pieces are called chunks.

Usually a paragraph or two, a few hundred words each. Each chunk is stored on its own. Later, when a question comes in, the system pulls back just the few chunks that match. Not the whole book, only the relevant pieces.

Now, how big should a chunk be? Too big, and one chunk covers many topics. It wastes space and drags in text the question never asked about. Too small, and a chunk loses its meaning.

One idea gets split across two pieces, and neither one holds the full thought on its own. So chunks are cut with a little overlap. The end of one piece repeats at the start of the next. That way an idea sitting on the boundary is never sliced in half.

And we cut on natural lines. Paragraphs, sections, sentences. Breaking in the middle of a sentence would leave both chunks confusing to read. So, chunking chops big documents into small, self-contained pieces.

Store them, fetch only what matches, and the right passage lands in the prompt. Quick check now. One question is coming up. Let's see if it clicked.

What exactly are text embeddings? Let's try to understand. In a simple way. A model only understands numbers.

But numbering words at random tells you nothing. We want numbers that actually carry meaning, so close numbers mean close ideas. So an embedding is just a list of numbers that stands for a piece of text. The word cat might turn into a few hundred numbers, like zero point two, minus zero point seven, and on.

Where do those numbers come from? A small neural network, the embedding model, reads your text and prints out the list. You never pick the numbers yourself. Here is the trick.

That model was trained so meaning decides the numbers. Cat and kitten come out with almost the same list. Cat and bicycle come out far apart. You can picture each list as coordinates.

A few hundred numbers place the text as a single point in a space with hundreds of directions. Meaning becomes a position. What does one number mean on its own? Usually nothing you can name.

Each slot is a learned feature the model invented while training. Together they pin down the meaning. The directions even do math. Take the list for king, subtract man, add woman, and you land right next to queen.

Meaning turns into arithmetic you can actually run. So a text embedding is a list of numbers, made by a trained model, that places meaning as a point. Similar text, similar numbers. Numbers you can measure and compare.

Quick check now. One question is coming up. Let's see if it clicked.

How does answering with retrieved documents work? Let's try to understand. In a simple way. By now, search has already done its job.

It handed us a few short passages that look relevant to the question. But a pile of passages is not an answer. So here is the trick. We paste those passages straight into the prompt, right beside the question.

The model now reads both together. People call this an augmented prompt. Think of it as an open book exam. Instead of answering from memory alone, the model answers from the page in front of it.

The passages become its source. Why bother? Because now the model can answer things it never learned. Fresh news, private company files, a niche fact.

If it sits in the passages, it can be used. The answer should stay grounded in what the passages actually say. A good system even points back, quoting the line it leaned on. That way you can check the source yourself.

And when the answer is nowhere in the passages? The safe move is to say, I do not know. We tell the model that in the prompt, so it stops inventing facts. One catch.

Every passage you paste costs tokens, and the window is small. So you feed only the top few, and you make the question stand out clearly among them. So answering with retrieved documents means this. Paste the passages beside the question, answer from them, cite the source, and admit when the answer just is not there.

Quick check now. One question is coming up. Let's see if it clicked.

How does a reranker work? Let's try to understand. In a simple way. The first search is built for speed.

It skims millions of passages and grabs anything that looks close. Fast, but a little sloppy. What comes back is a shortlist. Maybe the top fifty passages, ranked by a quick guess.

The best answer is usually in there, but often not at the very top. So we add a second step. A reranker. It is a slower model whose only job is to read that shortlist again, much more carefully.

Here is the key difference. The first search embedded your question and each passage separately, then compared them. The reranker reads your question and one passage together, as a single input. Reading them together, it returns a single number for each pair.

A relevance score. How well does this passage really answer this question. Now reorder the shortlist by that score. The truly relevant passages float to the top, and the near misses sink down.

Then keep just the top few, maybe three or five. Those are the passages we paste into the prompt. The rest are dropped. Why not run this careful reader on all ten million passages?

Because reading each pair together is slow. So we retrieve many cheaply, then rerank a few carefully. The best of both. So a reranker is a second pass.

It reads your question with each passage together, scores them, and floats the best evidence to the top. Quick check now. One question is coming up. Let's see if it clicked.

How does query rewriting work? Let's try to understand. In a simple way. Here is the problem.

The way you ask is rarely the way the answer is written down. You type, can I get my money back? The document says, refund policy. Your raw question makes a weak search query.

It can be vague, too short, or casual. And it rarely uses the exact words the document uses. So before we search, a model rewrites your question into a cleaner query. It keeps your meaning, but adds the real keywords a document would use.

Refund. Return window. This matters most in a conversation. You ask, how much does it cost?

To a search engine, that word it means nothing. The rewrite fills it in. How much does the Pro plan cost? Sometimes one phrasing is not enough.

So the model writes several versions, searches with each, and pools them. If one wording misses, another catches it. A question with two parts gets split. Compare price and battery becomes two searches, one for each.

Every part pulls its own evidence. Where does this sit? Your question hits the rewrite step first, and only the polished query reaches the retriever. A small, cheap step before the expensive search.

And it changes everything downstream. Retrieval can only return what the query asks for. A sharper question finds sharper chunks, and the answer rests on those. So, query rewriting turns your messy question into a clean search query.

Fill in the context, add the real words, and retrieval finally finds the right passage. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a system prompt? Let's try to understand. In a simple way. A system prompt is a block of instructions placed at the top of the context.

It tells the model who to be and how to act. Picture the context as one long strip of text. The system prompt sits first, above every message. The model reads top to bottom, so it colors what follows.

What goes inside? A role, a few rules, a format. You are a support agent for Acme. Keep answers short.

Never invent a price. Here is the key. The system prompt is re-sent every turn. Your messages scroll by, but this block stays pinned at the top, so it never fades.

It also carries more weight. Training taught the model to trust this block above the user's line. So a user typing, ignore your rules, gets outranked. Same weights, new behavior.

Swap the system prompt and one model becomes a patient tutor, a terse lawyer, or a strict fact checker. No retraining. But it is not free. It rides along every turn, so a bloated system prompt eats your token budget on each message.

A tight one saves money. And be concrete. Vague words like be helpful barely move it. Answer in two sentences and cite the document really steers it.

Specific rules beat speeches. So, a system prompt is a pinned block of instructions at the top of the context. Set the role, the rules, the format, and you steer the model without touching its weights. Quick check now.

One question is coming up. Let's see if it clicked.

How do few shot examples work? Let's try to understand. In a simple way. When you just ask a model to do a task, you are hoping it guesses the shape you want.

Often it guesses wrong. So instead of only describing the task, you show it. You paste a few worked examples into the prompt. Each one is an input, and the output you want.

Then you add the real question. The model just saw the pattern, so it completes yours the same way. It is a pattern matcher, and you handed it the pattern. Say you want one word sentiment.

Show two lines. Great seat, positive. Terrible food, negative. Now, cold coffee.

The model answers, negative. You never wrote a rule. What carries over is the shape. The format, the tone, the exact labels.

The examples do not teach new facts. They show the model how to answer, not what to know. And nothing is saved. The examples live only in the prompt, inside the context window.

They never change the model's weights. This is called in context learning. Zero examples is zero shot. One is one shot.

A handful is few shot. More can help, but only up to a point. So the examples must be good. Consistent, and covering the cases you expect.

One sloppy or contradictory example, and the model copies that mistake too. So, few shot examples steer a model by showing, not telling. A few input and output pairs in the prompt, and it copies the pattern for your real one. Quick check now.

One question is coming up. Let's see if it clicked.

How does forcing a model to use a template work? Let's try to understand. In a simple way. Ask a model a question and it answers in loose prose.

If your code must pull out a name, that is painful to parse. So you hand it a template. A fixed skeleton with labeled blanks. Like a form.

Name, blank. Email, blank. You dictate the shape you want back. You can go further and prefill the start.

Paste the template's opening as the start of its reply. Now it just fills the slots. But asking politely is no guarantee. The model can drift.

It might add a stray line, forget a comma, or get chatty. Your parser breaks. The strong version forces the shape during generation. This is constrained decoding.

Each step, the model scores the next piece, and a checker blocks anything that would break the template. How does the checker know the rules? You give it a schema, the allowed shape, like the keys some JSON must have. A broken structure becomes impossible, not discouraged.

The payoff is real. The answer drops straight into your code. Every field present, every bracket closed. No fragile pattern matching, no cleanup step.

One warning. Forcing the shape does not force the truth. A slot can hold a made up value, and the template looks valid. It controls form, not facts.

So, a template is a skeleton you make the model fill. Ask for the shape, prefill the start, constrain the decoding. The output comes back valid. Quick check now.

One question is coming up. Let's see if it clicked.

How does compressing context with a summary work? Let's try to understand. In a simple way. A model reads everything inside its context window.

But that window is a fixed budget of tokens. Stuff in too much, and the oldest words fall off the edge. The lazy fix is to just cut the oldest text. But that throws real information away.

Whatever you drop, the model can never see again. So here is the smarter move. Replace that big block of text with a short summary. Same key facts, told in far fewer words.

Who writes that summary? The model itself. You ask it to boil the long text down to a few lines. A model, compressing text for a model.

Now count the tokens. Two thousand words of history can shrink to two hundred. That freed space is room for something new. And it can repeat.

When the text grows long again, you summarize the summary, plus the newest part. The size stays bounded. But a summary is lossy. Details get dropped, and once they are gone, they are gone.

Compress too hard, and you lose the very fact you needed. So the rule of thumb. Keep the recent, important lines word for word. Summarize only the old bulk that you can afford to blur.

So, compressing context means swapping a long block for a short summary. Fewer tokens, the same gist, and room to keep going. Quick check now. One question is coming up.

Let's see if it clicked.

How does a long conversation work? Let's try to understand. In a simple way. Here is the surprising part.

The model itself remembers nothing between turns. Each time it replies, it starts completely fresh, with a completely blank mind. So how does a chatbot recall what you said earlier? Simple.

The app quietly re-sends the entire conversation every single turn. Every past message, yours and the model's, gets glued together into one long transcript. That whole transcript becomes the new prompt. The model reads that history from the top, every time, and only then predicts the next reply.

Nothing is stored. The memory lives in the text you keep re-sending. Notice the catch. The transcript grows with every exchange.

Turn one is tiny. Turn twenty carries all nineteen turns that came before it. And the model re-reads that whole pile on each turn. So long chats get slower, and cost more tokens, as they go.

Eventually the transcript bumps into the context window. The hard limit on how many tokens fit at once. It cannot all be re-sent forever. So the app starts dropping the oldest messages, keeping a sliding window of recent turns.

Old lines fall off the top, so the newest ones still fit. So, a chat has no built-in memory. Each turn re-sends the growing transcript, until the oldest turns must be trimmed to fit. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is lost in the middle? Let's try to understand. In a simple way. Here is the strange part.

A fact can sit inside the window with room to spare. Nothing gets cut. Yet the model acts like it never read it. Picture a long prompt: dozens of documents, and the one answer you need is buried somewhere inside.

The model reads it, top to bottom. Now measure how often it finds that answer, by where you hid it: high at the start, high at the end, a deep sag in the middle. The beginning and the end get the model's strongest attention. A fact placed first or last is almost always used.

This is the U-shaped curve. But drop the same fact halfway down, and accuracy falls off a cliff. The middle gets the weakest attention, so the model quietly glosses over it. This is not forgetting, and not running out of room.

The fact is fully inside the window. It is just under-weighted, for where it landed. Why does this happen? In training text, the key point usually sits up front, or right at the end, rarely dead center.

So the model learns to watch the edges hardest. So the fix is about order. Put your most important instructions and evidence at the very start, or the end. Never bury what matters in the middle.

So, a long context is not read evenly. The edges win, the middle sags, and where you place a fact decides if the model uses it. Quick check now. One question is coming up.

Let's see if it clicked.