LLMs in production

What exactly makes production LLMs uniquely fragile? Let's try to understand. In a simple way. Normal software is predictable.

Same input, same output, every time. When it breaks, it crashes loudly, with an error you can chase. A language model is different. Ask the exact same question twice, and you can get two different answers.

It samples its words, so randomness is baked right in. There is no single right answer. The output is free text, so you cannot simply check whether it equals the expected string. Good and bad sit on a blurry scale.

And it almost never crashes. When it is wrong, it does not throw an error. It hands you a fluent, confident sentence that happens to be false. You also did not write the model.

It is a black box behind an API. The provider can update it underneath you, so yesterday's prompt may behave differently today. The input is open-ended language. Users can type literally anything, so the space of inputs is endless.

You can never test every case in advance. The prompt itself is brittle. Reword one line, or add one example, and the behavior can swing hard. A tiny edit is never truly small.

Now stack it up. Random outputs, no clear pass or fail, silent mistakes, a shifting black box, endless inputs. Ordinary testing assumes none of that. Production LLMs are fragile because they are random, hard to grade, quiet when wrong, and built on a model you do not control.

That is why they need a new playbook. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is an evaluation set? Let's try to understand. In a simple way. When you ship an AI feature, everyone agrees it should be good.

But good is a feeling, not a number. You cannot improve what you cannot measure. So how do most people check? They tweak the prompt, try a couple of examples by hand, it looks fine, they ship.

That is just a vibe check. The fix is an evaluation set. A fixed list of example inputs, each paired with what a good answer looks like. Your definition of good, written down.

Now push every example through the model and grade each answer. Count how many pass. Good stops being a feeling. It becomes a single number you can track.

What goes in? Real cases. The everyday requests, the tricky edge cases, and every bug a user has ever reported. If it matters to you, it belongs here.

And you freeze it. The same inputs, every run. So when you change a prompt or a model, you compare the new score against the old, fairly. Bigger is not better here.

Fifty sharp, hand-picked examples beat ten thousand random ones. Each entry should test something you genuinely care about. Here is the payoff. A change that fixes one case but quietly breaks three others now shows up as a lower score.

Silent regressions cannot hide. So an evaluation set turns the fuzzy word good into a scoreboard. Representative inputs, the answers you expect, and one honest number to chase. Quick check now.

One question is coming up. Let's see if it clicked.

How does scoring outputs with semantic similarity work? Let's try to understand. In a simple way. You have an evaluation set, with a trusted reference answer for each question.

The model replies in free flowing text. Now grade thousands of these, automatically. The easy way is to compare the two texts letter for letter. But that breaks fast.

The reference is, Paris. The model says, the capital is Paris. Both are correct, yet exact match calls it wrong. So we need to judge the meaning, not the spelling.

Two answers can use different words and still say the very same thing. Here is the trick. An embedding model reads a sentence and turns it into a vector. A long list of numbers that captures what the sentence means.

Picture each vector as an arrow in space. Two sentences that mean the same thing point the same way. Different meanings point apart. So we measure the angle between the two arrows.

This is called cosine similarity. Near one means same direction, same meaning. Near zero means unrelated. Then you pick a cutoff.

Say zero point eight. Score above it, the answer passes. Below it, it fails. Suddenly free flowing text can be graded automatically, at scale.

One warning. This scores similarity, not truth. A fluent, confident wrong answer can still sit close to the reference. So never trust it alone.

So, to score by meaning. Embed both answers into vectors. Compare their directions with cosine similarity. Then threshold the score into a pass or a fail.

Quick check now. One question is coming up. Let's see if it clicked.

How does running evals in your CI pipeline work? Let's try to understand. In a simple way. You already have an evaluation set that turns quality into a score.

But running it by hand is fragile. One busy day, someone forgets. The fix is to make it automatic. Your CI pipeline is a robot that watches your code.

Every push or pull request wakes it up to run a checklist. So you add one more step to that list. Beside the usual tests, the pipeline runs your whole evaluation set, grading every example on its own. Then you set a bar.

The suite hands back a score. At or above the bar, the check turns green. Below it, the check fails, in red. And a failed check blocks the merge.

The change cannot reach production until the score climbs back. That red mark is a gate in the way. But the model is random, so one run can wobble. You do not demand a perfect score.

You run each case a few times and ask for a pass rate. Every check costs money and time, since each example calls the model. So keep the set small and sharp, and run it on every pull request. Now nobody has to remember.

Every change gets the same test, the same way, every time. A quiet break now turns the pipeline red before it ships. So, running evals in CI makes your quality bar automatic. Every change is graded, and only the ones that clear the bar get through.

Quick check now. One question is coming up. Let's see if it clicked.

How does versioning prompts, models, and data together work? Let's try to understand. In a simple way. An AI feature is more than code.

Its answer depends on three moving parts: the prompt you send, the model you call, and the data you feed in. Change any one of those three, and the output shifts. A reworded prompt, a newer model, one swapped example. The same input can give a different reply.

So when something breaks in production, the question is brutal. Which prompt, which model, which data made that answer? If you never wrote it down, you are guessing. The fix is to treat all three as one bundle.

Freeze the exact prompt, model version, and examples. Then stamp that bundle with a version number. Think of it like git, but for your whole setup, not just the code. Change the prompt, the model, or an example, and you cut a fresh version.

Now every output carries the version tag that made it. See a strange answer? Load that exact bundle and reproduce the result yourself. This matters because the model is rented, not yours.

The provider can quietly update it. Pinning the model version keeps yesterday's behavior from vanishing overnight. And since a version locks all three, comparing version seven with version eight is fair. Only what you changed differs, so you can find the real cause.

So versioning ties prompt, model, and data into one numbered bundle. Every result traces to a version. That is what lets you reproduce, compare, and trust what shipped. Quick check now.

One question is coming up. Let's see if it clicked.

How does deploying new versions with shadow mode work? Let's try to understand. In a simple way. Your new version passed offline tests.

But those only cover cases you imagined. Real traffic is messier, and never ends. You want it tried on real users, no bad answer reaching anyone. Shadow mode runs two versions at once.

The live one, that every user already gets today. And the new candidate, the challenger you want to test. Every incoming request gets copied. Both versions receive the exact same input, at the same moment.

The new one runs quietly in the shadow of the old. Here is the trick. Only the live version's answer reaches the user. The candidate's answer is logged, then quietly discarded.

Nobody out there feels a thing. But you just captured something valuable. The new version's real outputs, on real inputs. Now compare.

Is it slower, pricier, do the answers hold up? And because this runs on live traffic, it surfaces the strange inputs your offline set never imagined. A dress rehearsal on a real stage. There is a catch.

The shadow must stay read only. It cannot send an email, charge a card, or write to the database. And you pay for two calls. When the shadow holds up across enough real traffic, you flip the switch.

The candidate becomes the live version. A calm, evidence backed release. So shadow mode mirrors real traffic to the new version, serves only the old one, and compares in secret. You test against reality, users feel nothing.

Quick check now. One question is coming up. Let's see if it clicked.

How does logging key metrics from every LLM call work? Let's try to understand. In a simple way. Remember, a language model can fail quietly.

It does not crash. It hands you a wrong answer. So you are blind, unless you record what happened on every call. The fix is simple.

For each call, save one structured record. Think of it as a single row in a table. One row for every request. First, log the inputs.

The exact prompt you sent, and the model and version you called. This is what lets you reproduce the call later. Next, log the result. The text the model returned, and a status flag.

Did the call succeed, or did it error out? Now the performance numbers. Latency first. How many milliseconds the call took.

This is how you catch the model getting slow, before your users complain. Also log the token counts. How many tokens went in, and how many came back. These two numbers drive your speed and your bill.

Here is the key rule. Log every call, not a sample. The one you skip is the one that quietly broke. Rare failures only appear if you catch them all.

Now the payoff. Thousands of rows become a dashboard. Average latency, the slowest one percent, the error rate over time. Raw logs turn into a live health check.

So, wrap every model call in a logger. Capture the inputs, the output, the latency, and the tokens, every single time. Those records are your only eyes in production. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is concept drift in prompts? Let's try to understand. In a simple way. You wrote a prompt, tested it, and shipped it.

Nothing on your side changes after that. The prompt is frozen, the model is frozen, and it works great. But the world keeps moving. The questions your users type slowly change.

New topics appear, new slang, a new feature they suddenly ask about. That slow shift has a name. Concept drift. The pattern behind the inputs has moved, so a prompt tuned for yesterday quietly stops fitting today.

And it is silent. There is no error and no crash. Quality just slips a little each week, while nobody watches the trend. So how do you catch it?

First, take a baseline. Back when things looked good, snapshot your inputs and your quality score. That is your reference point. Then watch the inputs.

Are new words and topics showing up that your baseline never saw? A rising share of unfamiliar inputs is an early warning sign. Also track a quality signal over time. The thumbs down rate, the fallback rate, or a score on fresh samples.

Plot it week by week, not just once. Now compare. Line up a recent window against the baseline. A widening gap is the drift.

Set a threshold, and let it raise an alarm. So concept drift is the world moving underneath a frozen prompt. You catch it by watching your inputs and quality drift away from a baseline, and alerting on the gap before users feel it. Quick check now.

One question is coming up. Let's see if it clicked.

How does capturing user feedback on model responses work? Let's try to understand. In a simple way. Your model is live, answering real people.

But is it any good? Your offline scores were frozen the day you shipped. The users on the other end are the ones who actually know. So you ask them.

The simplest way is a thumbs up and a thumbs down button, right next to every answer. That is explicit feedback, given on purpose. But a lonely thumbs down is useless. So you glue each vote to the exact request.

The input, the answer, and which prompt version and model made it. Here is the problem. Almost nobody clicks the button. So instead, watch what people do.

Did they copy the answer, edit it, retry, or just leave? Those actions are implicit feedback. And the few clicks you get are skewed. An angry user reaches for thumbs down far more often than a happy one taps thumbs up.

So raw counts lie. Two fixes. Make the vote a single tap so more people bother. And lean on the quiet implicit signals to fill the silent majority's gap.

Now the payoff. That growing pile of judged answers lets you cluster the failures, feed your evaluation set, and even serve as preference data to fine tune the model. So capturing feedback means catching both signals, the explicit thumbs and the implicit actions, gluing each to its request, and turning real reactions into your next improvement. Quick check now.

One question is coming up. Let's see if it clicked.

What makes a safe rollback possible? Let's try to understand. In a simple way. You shipped a new version, and it is live and misbehaving.

The answers got worse, or the cost spiked. You need a way out, now. That way out is a rollback. You return to the previous version, the one you know worked.

Not a fix, a retreat. First, the old version still exists. Every release was saved as a complete bundle: prompt, model, and config. The good one you replaced is still there, untouched.

Second, the switch is just a pointer flip. You do not rebuild anything, you point traffic back at the old bundle. One config change, seconds, not hours. Now the part people forget.

If, going forward, the new version rewrote the database or dropped the old data format, the old version has nothing to read. That is a one way door. Forward changes must stay reversible. You can only roll back fast if you notice fast.

A threshold on your live metrics is crossed, an alert fires. That is the signal to flip. And do not improvise it. Rehearse the rollback before you need it.

A single tested button, or an automatic flip when a health check fails. A rollback is not the repair. It puts users back on safe ground, and buys you calm to debug the broken version offline. So a safe rollback rests on four things.

The old version kept intact. A one flip switch. Forward changes that stay reversible. And a fast alarm to pull it.

Quick check now. One question is coming up. Let's see if it clicked.

How does response caching work? Let's try to understand. In a simple way. Every call to a large model is slow and costs money.

It can take a second or more, and you pay per token. At scale, that adds up. Here is the key thing. In production, the same questions come back again and again.

So why pay the model to answer the same thing twice? So you put a cache in front of the model. A cache just stores past prompts and their answers. Ask once, save the pair, reuse it later.

Now every request checks the cache first. If the prompt is already there, that is a hit. You return the saved answer in milliseconds, with no model call. If the prompt is not there, that is a miss.

The request falls through to the model. You store that fresh answer, so the next identical request hits. The simplest cache keys on the exact text. Same prompt in, same answer out.

But change one word, or even one space, and it misses. A semantic cache fixes that. It turns each prompt into a vector, an embedding, then finds a near neighbor. Similar meaning is a hit, so paraphrases share one answer.

But watch for staleness. If the real data changes, a cached answer is now wrong. So give each entry a time to live, and drop it once it expires. So caching trades a little memory for big savings.

Check first, serve hits for free, and only pay the model on a miss. Quick check now. One question is coming up. Let's see if it clicked.

How do guardrails against unsafe outputs work? Let's try to understand. In a simple way. A language model is probabilistic.

No matter how careful your prompt, it can still say something toxic, leak a private detail, or go off the rails. On live traffic, that happens. A guardrail is a separate checking layer wrapped around the model. It inspects what goes in, and what comes out, before anything reaches a real user.

First, the input rail. Before the model even runs, it screens the incoming message. Is this a jailbreak, a prompt injection, or wildly off topic? If so, stop early.

Then, the output rail. The model answers, but that answer is held back and inspected first. Is it toxic, does it leak private data, is the format wrong? These checks are not the model itself.

A simple rule or blocklist, a small classifier that scores toxicity, a schema validator, or a second model as a judge. When a check trips, you have choices. Block the answer, redact the private parts, or swap in a safe, canned reply. The unsafe version never ships.

This is different from your offline tests. Those run once, before release. Guardrails run live, on every request, catching the bad output in the moment. There is a price.

Every rail adds a little latency, and sometimes another model call. But it is the seatbelt that stops a bad answer reaching a user. So a guardrail wraps the model with checks. Screen the input, screen the output, block anything unsafe before it ships.

Quick check now. One question is coming up. Let's see if it clicked.

How does tracing a single request work? Let's try to understand. In a simple way. One user request is almost never one step.

It fans out: retrieve documents, build a prompt, call the model, maybe run a tool, then clean up the answer. Now one answer, out of thousands, comes back wrong or slow. Which step broke it? Your logs are scattered lines, with nothing tying them together.

The fix starts with one shared label. The moment the request arrives, it gets a trace id. That same id is carried to every step it touches. Then each step opens a span.

A span is one unit of work, timed. It records the step's name, when it began, and how long it took. And spans nest. The step that kicks off another is the parent, the one it calls is the child.

That link records who triggered what. Now stitch every span that shares a trace id. They line up into one waterfall. One bar per step, on a time axis, so you see where the time went.

So the bad request is now one click. Filter by its trace id, open the waterfall, and the slow or failing span jumps out. Retrieval took four seconds. One more thing.

A span can carry details too: the exact prompt, the model version, the retrieved chunks. So you replay what happened, not guess. So, tracing a request rests on three things. One shared id, carried everywhere.

A span timing each step. And a waterfall that shows exactly where it broke. Quick check now. One question is coming up.

Let's see if it clicked.

How does tracking token costs work? Let's try to understand. In a simple way. First, the money.

A provider does not bill you per request. It bills you per token, every token in and every token out. And two meters are running. Input tokens, the prompt you send.

Output tokens, the answer it writes back. Output usually costs more per token. Here is the good news. Every call hands you the count.

The response carries a usage field: prompt tokens, completion tokens, and a total. No guessing. So the cost of one prediction is simple. Input tokens times the input price, plus output tokens times the output price.

That sum is the call's bill. Prices are tiny, quoted per thousand or per million tokens. One call is a fraction of a cent. But run it a million times a day, and those fractions add up.

Now the important move. Tag every call in your logs. Which user, which feature, which model. Without tags, you just have one giant invoice.

With the tags, you can add the costs up any way you like. Cost per user, per feature, per day. The expensive corners of your product show themselves. And this is how you catch trouble early.

A prompt that doubled in length. One user hammering the system. Set a budget, and an alert fires before the invoice does. So, track the tokens on every prediction.

Read the usage, turn it into dollars, tag it. That is how a token bill stops being a surprise. Quick check now. One question is coming up.

Let's see if it clicked.

How does a local prompt become a production endpoint? Let's try to understand. In a simple way. Right now your prompt lives in a script on your laptop.

You run it by hand, and read the answer yourself. It works, but only for you. A real app cannot phone your laptop. Users need this always on, and callable by other software, not by a person typing.

So here is the key move. You wrap the prompt in a small web service. Now it is an endpoint. One web address that takes a request, and returns a response.

That endpoint is a contract. Callers send input in, and get an answer back, in a fixed shape. They never see your prompt, the model, or your keys. Next, the service leaves your laptop.

It runs on a server that stays up day and night. So the endpoint is reachable even while you sleep. Your secret key moves too. It lives on the server now, never in the client.

The caller hits your endpoint, and your server quietly calls the model. Production means many callers at once. So you keep each request self contained, holding no memory between calls. Then run several copies behind one shared address.

And the endpoint is where everything else attaches. Logging, caching, guardrails, versioning, a rollback button. The same prompt, now wrapped in a safety net. So, from laptop to endpoint.

Wrap the prompt in a service, deploy it to a server, hide the key, and let any app call it. Quick check now. One question is coming up. Let's see if it clicked.