LLM guardrails in production: a working architecture

A single content filter in front of a model is a speed bump, not a guardrail. Real protection for a customer-facing assistant is a chain: independent layers, each catching what the layer before it structurally cannot see, arranged so the cheapest and highest-stakes checks run first. This is the architecture we run on a live shopping assistant in a regulated category (supplements), generalized so it ports to any store where an AI feature can say something wrong in public.

Why one filter isn't enough

Different failures live at different points in the request. An unsafe topic (a medical or self-harm question) has to be caught before the model spends a token. An off-topic hijack is a routing problem. A leaked competitor recommendation or a regulated health claim only exists after generation. A cost blowout is about the shape of the whole conversation. No single check sees all of these, so you place a specific defense at each point and let them compound. Defense in depth is not paranoia here; it is the only design that matches where the failures actually occur.

Before any model call: the deterministic pre-classifier

The first and most important layer never invokes the model. A fast, deterministic classifier checks the incoming message against a curated list of topics that must never reach a generative model unfiltered — in our category: eating disorders, advice aimed at minors, medication interactions, pregnancy, and banned substances. A match short-circuits to a safe, supportive, human-written response before the main model sees a single word. It is the cheapest layer and the one with the highest stakes, which is exactly why it runs first and runs deterministically — you do not want the reliability of a safety boundary to depend on a probabilistic model's mood.

Scope, sanitation and input defenses

Two more layers act on the way in. A scope classifier keeps a shopping assistant on the subject of shopping: off-topic requests get a friendly redirect, which also neutralizes most prompt-injection attempts — an injected instruction accomplishes little when the whole turn is classified out of scope and never reaches the tool loop. Then input sanitation strips markup, caps lengths, and normalizes the strange things people paste. These are unglamorous, but they remove an entire class of attacks cheaply.

A design note that matters more than it looks: each layer should be independent, so a bypass of one does not cascade. If your scope classifier and your output validator share the same model call and the same failure mode, you have one guardrail wearing two hats, not two guardrails. Deterministic checks where determinism is possible (the safety list, sanitation, PII patterns) and model-based checks only where judgement is genuinely required — that split keeps the chain both cheap and robust, because the layers most likely to be attacked are the ones that don't depend on a model at all.

Output validation: the last line

After generation and before the customer sees anything, the output passes a battery of validators: the category claim validator above, a brand-leak scanner so the assistant never sends a shopper to a competitor's store, and prompt-extraction detection that catches attempts to make the bot reveal its own instructions on the way out. Output validation is where you enforce the promises grounding can't: grounding decides what facts the model may use; output validation decides what it is allowed to say. It pairs naturally with the tool-grounding discipline in RAG over product data and the schema discipline in structured LLM outputs.

Privacy and economic guardrails

Two boundaries protect the business rather than the answer. Privacy plumbing redacts PII before anything is logged or forwarded to analytics, and a retention clock purges conversations on schedule — GDPR as a design input, not a review at the end (the same first-party, consent-gated posture as our analytics consent handling). Economic guardrails are a per-request cost ledger, per-session rate limits, and a capped tool-call loop: a model asking for a fifth round of tool calls is lost, not thorough. An assistant with no cost ceiling is a denial-of-wallet endpoint waiting to be found.

Typed refusals: decline as a first-class response

When any layer declines, the refusal is not an apology paragraph — it is a typed response with a category, and the UI renders each category differently. A safety refusal gets a supportive message and, where appropriate, a pointer to a professional; an off-topic refusal nudges back toward what the assistant can do; a rate-limit refusal is a plain "try again shortly". Two hard-won rules: standardize the envelope — we once had two refusal shapes emitted by two code paths and a frontend that only handled the one it was tested against — and remember that refusal quality is brand quality, because the customer who hits a guardrail is disproportionately vulnerable, frustrated, or probing you. The whole chain is worth nothing if the moment it fires reads as broken. See how these refusals slot into the two assistant shapes in the assistant architecture, and how prompt changes to any of this ship safely in prompt management.

Frequently asked questions