What exactly is retrieval? Let's try to understand. In a simple way. Here is the thing.
Everything a model knows is baked into its weights during training. After that, the weights are frozen. Nothing new gets in. So there is a cutoff date.
Ask about yesterday's match, or a product launched last week, and the model simply has no way to know. Same story with private stuff. Your company's contracts, your internal wiki, your emails. None of that was in the training data.
The model has never seen it. And when a model does not know, it rarely says so. It predicts the most plausible next words. That is how you get confident, fluent, wrong answers.
Hallucination. So here is the fix. Before answering, go fetch the relevant text. Paste it into the prompt.
Now the model answers with the facts right in front of it. Like an open book exam. The literature has names for this. Knowledge in the weights is parametric memory.
Knowledge fetched from outside is non-parametric. Retrieval augmented generation, RAG, combines both. This also grounds the answer. The model can quote the passage, and you can check the source.
Trust goes way up. And updates get cheap. New facts? Just change the documents.
No retraining, no new weights. The knowledge store and the model live separate lives. So, models need retrieval because their built-in knowledge is frozen, incomplete, and unverifiable. Fetch the facts at question time, and let the model do the reasoning.
Quick check now. One question is coming up. Let's see if it clicked.
What exactly is an embedding? Let's try to understand. In a simple way. Here is a problem.
You search for car repair. The document says automobile maintenance. No word matches, so keyword search fails. But the meaning is the same.
The trick is to turn text into a point in space. An embedding is a long list of numbers, a vector. One sentence in, one vector out. And here is the magic.
Similar meanings land close together. Doctor sits near nurse. Far away from banana. Meaning turns into geometry, and distance becomes a measure of relatedness.
So back to our search. Those two phrases share zero words, but their vectors land almost on top of each other. Search by distance, and you catch matches keywords miss. Now, this space is not flat like a map.
Real embeddings use hundreds or thousands of dimensions. Say fifteen hundred numbers per sentence, each capturing some shade of meaning. How close is close? The standard measure is cosine similarity.
It looks at the angle between two vectors. Small angle, same direction, similar meaning. Where does the map come from? An embedding model, a neural network trained on millions of text pairs.
Related pairs get pulled together, unrelated pairs get pushed apart. That is contrastive learning. And this is the engine inside RAG. Embed the question, embed the documents.
The relevant passages are simply the nearest points. So, an embedding is meaning written as coordinates. Close means related, far means unrelated. Machines can finally compare ideas, not just words.
Quick check now. One question is coming up. Let's see if it clicked.
How does a vector database work? Let's try to understand. In a simple way. So you have turned your documents into embeddings.
Millions of vectors. Now a question comes in, also as a vector. The job: find the vectors closest to it. The honest way is brute force.
Compare the query to every single vector, one by one. With a million vectors, that is a million comparisons. Per question. A vector database skips most of that.
Think of a library that groups similar books on nearby shelves. You walk to the right shelf, and ignore the rest. The real mechanism is called approximate nearest neighbor search. ANN.
The database builds an index, a structure that lets it check only a tiny slice of the vectors. The most popular index is HNSW. A graph of vectors, where each one links to its neighbors. Search hops along links, getting closer with every hop.
Here is the catch. Approximate means approximate. Sometimes the true closest match gets skipped. You trade a little recall for a huge amount of speed.
And it stores more than vectors. Each one carries its original text and metadata. So you can filter, say only documents from this year, then search inside. You ask for the top few matches.
Say the ten nearest vectors. The database returns them with their text, ranked by similarity, in milliseconds. So, a vector database is a search engine for meaning. It indexes vectors, finds nearest neighbors fast, and hands back the text they came from.
Quick check now. One question is coming up. Let's see if it clicked.
What exactly is chunking? Let's try to understand. In a simple way. In RAG, you don't retrieve whole documents.
You retrieve small pieces. So before storage, every document is cut into chunks. And how you cut decides what retrieval can find later. Here is the thing.
Each chunk gets compressed into one embedding. A single vector that has to carry the meaning of the entire piece. Cut too big, and that vector turns muddy. A ten page chunk about refunds, shipping, and privacy averages into a blur.
Your refund question won't match it well. Cut too small, and you lose context. A lone sentence like, it takes thirty days, matches nicely. But thirty days for what?
Nobody knows anymore. The classic recipe is fixed size chunking. Say five hundred tokens per chunk. Plus overlap, maybe fifty tokens repeated between neighbors, so no sentence gets sliced in half at a boundary.
Better still, cut where the author did. Recursive splitting tries headings first, then paragraphs, then sentences. It only falls back to raw size when it has to. There is also semantic chunking.
You embed each sentence, walk through them in order, and cut wherever the meaning shifts. Costlier, but every chunk stays on one topic. In practice, a few hundred tokens is the usual sweet spot. Big enough to answer a question on its own.
Small enough to stay about one thing. So, chunking is deciding what one vector should mean. Cut on natural boundaries, keep a little overlap, and keep each chunk about one idea. Quick check now.
One question is coming up. Let's see if it clicked.
How does the RAG pipeline work? Let's try to understand. In a simple way. Here is the thing.
RAG is not one step. It's a pipeline with two halves. One half runs ahead of time. The other runs the moment you ask a question.
Ahead of time, you prepare the library. Documents get cut into chunks, each chunk gets an embedding, and everything lands in a vector database. Done once, ready forever. Now the live half.
You ask a question. First move: your question is embedded too. Same model, same space as the chunks. Then the search.
The database finds the chunks whose vectors sit closest to your question. Say the top five. Those come back as plain text. Next comes augmentation.
The system builds a new prompt. Your question, plus the retrieved chunks, plus one instruction: answer using only this context. Finally, generation. The LLM reads that stuffed prompt and writes the answer.
It never searched anything. It just answered from what was pasted in front of it. Watch it run. You ask, what's our refund policy?
The question becomes a vector, three policy chunks come back, and the model replies: thirty days, receipt required. Grounded. Two details matter. Use the same embedding model on both sides, or the geometry breaks.
And keep the sources, so the answer can cite them. So that's the pipeline. Index once. Then embed the question, retrieve the neighbors, augment the prompt, generate the answer.
Retrieval, augmentation, generation. That's the name. Quick check now. One question is coming up.
Let's see if it clicked.
How does reranking and hybrid search work? Let's try to understand. In a simple way. Plain vector search has a weakness.
It finds passages that are close in meaning. But close is not the same as best. The first pass is a rough cut. And embeddings blur details.
Search for error code four oh four, or a part number, and semantic search may miss the exact string. Keyword search nails it. So hybrid search runs both. A keyword ranker, usually BM25, plus the vector search.
Two lists, two kinds of evidence. Then you merge them. A common trick is reciprocal rank fusion. A document that ranks high on either list floats to the top of the combined one.
Now, reranking. Take the top fifty candidates, and let a stronger model read each one next to the question. Then reorder them by that score. That stronger model is a cross encoder.
It sees the query and the passage together, in one pass, so it can catch fine details that separate embeddings miss. So why not cross encode everything? Too slow. It must run once per document, per query.
A million documents would take forever. Embeddings are precomputed, so they search in milliseconds. So the shape is a funnel. A cheap, wide net pulls in candidates fast.
An expensive, careful judge reorders the shortlist. Speed first, precision second. So, hybrid search widens what you catch, keywords plus meaning. Reranking sharpens the order before the LLM reads it.
Better context in, better answers out. Quick check now. One question is coming up. Let's see if it clicked.
What exactly is a RAG failure? Let's try to understand. In a simple way. Here is the thing.
RAG is a chain. Retrieve, then read, then answer. If any link breaks, the final answer breaks. So let's look at each link.
First failure: the retriever misses. You ask about sick leave, but the policy says medical absence. Different words, distant vectors, so the right chunk never comes back. And here is the scary part.
The model does not say, retrieval failed. It writes a fluent, confident answer from thin air. Grounding gone, hallucination back. Second failure: the answer lives nowhere.
Ask, how many clients complained this year. That needs every complaint, counted. No single chunk holds that, so top five chunks can't either. Same story for multi-hop questions.
Who manages the engineer who wrote this doc? You need one fact to find the next. One round of retrieval only makes one hop. Third failure: the model ignores what you gave it.
Stuff ten chunks in, and facts buried in the middle get skipped. Researchers call this lost in the middle. Fourth failure: retrieval works, but the documents lie. A stale policy from two years ago, or two files that contradict each other.
The model grounds itself in garbage. So RAG fails four ways. Miss the chunk, need what no chunk holds, ignore the middle, or trust bad documents. Fixes exist, but diagnosis comes first.
Quick check now. One question is coming up. Let's see if it clicked.
What exactly is the RAG, long context or fine-tuning decision? Let's try to understand. In a simple way. You have documents your model has never seen.
There are three ways in. Retrieve them, paste them all, or train them into the weights. Long context is the simple one. Paste everything into the prompt and let attention find what matters.
It works, but you pay for every token, on every call. RAG fetches just the relevant pieces at query time. The knowledge lives outside the model, so you can update it tonight and it's live tomorrow. Fine-tuning goes deeper.
You keep training the model on your data, and the knowledge gets baked into the weights themselves. That's called parametric knowledge. Here is the thing. Fine-tuning is great at teaching behavior.
Style, tone, output format. But it's unreliable for facts, and updating one fact means training again. Fine-tuning also carries a risk called catastrophic forgetting. Push new knowledge in too hard, and the model can lose skills it already had.
Picture a law firm with a million contracts that change daily. Long context can't hold them. Fine-tuning can't keep up. RAG retrieves the three contracts that matter, with sources.
So when does long context win? When the material is small, used once, and needs full reading. Like one report you want summarized end to end. And it's not either or.
Real systems combine them. Fine-tune for behavior, retrieve for facts, and use the long window to hold what retrieval brings back. Quick check now. One question is coming up.
Let's see if it clicked.