Evaluating LLM features in production: corrections, judges and calibration

Offline benchmarks tell you how a model does on someone else's test set. They tell you almost nothing about whether your feature is getting better for your shoppers. Production evaluation is a different craft: it runs on the behavioral signals your users generate for free, a stronger model acting as a blind judge, and a calibration check that catches the failure benchmarks miss. Here is the evaluation loop we run on a live shopping assistant.

Log everything, fire-and-forget

Evaluation starts before you have anything to evaluate: log every recommendation with its full inputs — the query, the retrieved candidates, the tools called, the final answer — written fire-and-forget so it never blocks or slows the customer's request. The point is to turn quality analysis into a query over history rather than a debate over anecdotes. If you can't reconstruct exactly why the assistant said what it said three weeks ago, you can't tell whether last week's change helped. Log first; you cannot retrofit a decision trail. This pairs with the query-to-click instrumentation in site search analytics — same philosophy, different surface.

User corrections are free labels

The best labels in the building are the ones your users produce without being asked. Every correction, edit, abandoned plan, and item swapped out of a proposed set is a labeled example of the assistant being wrong — collected continuously, at zero labeling cost, from exactly the distribution you care about. Hand-written eval sets go stale and drift from real usage; behavioral labels never do, because they are real usage.

Two cautions: not every abandonment is a correction (people get interrupted), so weight noisy signals accordingly; and make the correction easy to give — an assistant that's hard to correct simply loses the data.

A stronger model as a blind judge

Corrections tell you when you were wrong; they don't tell you how close you were to a better answer. For that, periodically have a stronger model re-answer the same requests blind — without seeing the production answer — and compare. This surfaces systematic drift a dashboard won't, and it turns the judge into a preference signal you can track over time. Keep the judge blind (bias toward whatever it sees first is real), sample rather than judging everything (it's expensive), and audit the judge occasionally against a human — a judge you never check is just another unverified model.

The judge is most valuable on the answers a correction never touches — the ones the shopper accepted but that a better model would have improved. Corrections only flag the failures obvious enough for a user to react to; the judge catches the quiet mediocrity underneath, the answers that were "fine" but not right. Together they cover both ends: user behavior for the errors that cost you conversions today, the judge for the ceiling you're leaving on the table.

Calibration, not just accuracy

Accuracy asks "was the answer right". Calibration asks "did the confidence match the reality", and for anything that carries a confidence signal — a goal-fit score, an "I'm fairly sure" hedge — calibration is the metric that decides whether that signal is worth showing. The test is simple: do the answers the assistant marked confident get corrected less often than the ones it marked uncertain? If they're corrected at the same rate, the confidence signal is decorative and any UI built on it is quietly lying to the shopper. Calibration is where honest uncertainty (see structured LLM outputs) gets verified rather than assumed.

When to upgrade the model (and when not to)

The blind judge doubles as an upgrade oracle, and this is where it saves real money. If a more capable model, judging blind, only marginally beats your production model, the upgrade isn't worth its added latency and cost yet — the gap doesn't reach the shopper. If it wins consistently and decisively on the request types that matter, that's your signal. Either way the decision is measured, not vibes-driven, which also feeds the routing decisions in LLM cost optimization: you don't pay for a bigger model until the judge proves it earns its keep.

Offline evals vs production truth

Offline evaluation still has a job — it's the fast, cheap gate that catches gross regressions before a change reaches anyone, and it's where you replay your accumulated corrections as a regression suite. But treat it as a filter, not the verdict. The verdict comes from production behavior, because the only distribution that matters is the one your shoppers actually generate. Run offline evals in CI to fail fast; run the behavioral loop in production to know the truth. And ship every change to the prompts behind them the way prompt management describes — a canary, a bake, automatic rollback — so evaluation and deployment reinforce each other instead of fighting.

One caution about the regression suite: prune it. A corrections-derived test set grows without bound, and old cases stop reflecting current traffic or a since-fixed bug. Periodically retire cases that no longer represent real usage, and weight the suite toward the intents and languages that actually drive revenue, so a green run means "we didn't break what matters" rather than "we passed a thousand stale checks". Evaluation is a product with its own maintenance cost; budget for it, or it slowly stops telling you the truth.

Frequently asked questions