LLM Guardrails: Input and Output Validation
On this page
Learning Objective
Design layered input and output controls around a model, decide where deterministic validation beats model-based classification, explain why guardrails must fail closed, and place validation correctly relative to streaming and to any action the output triggers.
Concept Explanation
Guardrails are the controls that sit between the model and everything else, and the useful way to think about them is as a specific application of a principle the security lessons of this course already established: an untrusted component's output must be validated before it is trusted, and an untrusted component's input must be constrained before it is accepted. What makes this harder than ordinary input validation is that the model is both a producer and a consumer of unbounded natural language, so neither side has a simple grammar to check against. What makes it tractable is that most of what needs checking is not actually about natural language at all.
That last point is the most practically valuable one in this lesson. Teams reach for a model-based classifier to check outputs when a deterministic rule would be faster, cheaper, more reliable, and testable. Does the response parse against the schema. Are all cited document identifiers present in the retrieved set. Does the answer contain a URL outside the allow-list. Does it contain something matching a credit card or national identifier pattern that should have been redacted. Is it within the length bound. Does the proposed tool call name a permitted tool with arguments of the right type and within permitted ranges. Every one of those is a deterministic check, and collectively they catch a large share of real incidents. Model-based classification should be reserved for the genuinely fuzzy judgments, principally safety categories and topic boundaries, where no rule can express the criterion.
Input controls come first and are cheaper than output controls because they prevent work rather than discard it. Bound the input length before assembly so an oversized document cannot inflate cost or push out retrieved context. Rate limit per authenticated principal, as established earlier. Redact or tokenize personal and regulated data before it leaves your boundary, which is a compliance control independent of any model behavior. Run a topical scope check when the product has a defined scope, so an off-topic request is declined cheaply rather than answered expensively. And apply injection-pattern screening as one layer among several, understanding from the next lesson that it is a filter rather than a solution.
Output controls need to be placed correctly relative to two things, and both placements are frequently wrong. The first is streaming: as the streaming lesson established, once the first token has been sent you cannot retract it, so a guardrail either operates on a sliding buffer held back from the user or the content is not streamed at all. The second and more important placement is relative to action. If the model output triggers anything - a tool call, a database write, an email, a payment, a code execution - validation must occur before that action, and it must be a hard gate rather than an advisory score. This is the boundary where a hallucination stops being a bad answer and becomes a bad action, and it is the single most important place in the entire system to be strict.
Finally, guardrails must fail closed, which connects directly to the exceptional-conditions material in this repository's security course. If the moderation service times out, if the validator throws, if the schema library errors, the safe response is to block or degrade rather than to pass the content through unchecked. The common anti-pattern is a try-catch around the guardrail that logs a warning and continues, which means the control silently disappears exactly when the system is under stress and least able to tolerate a bad output. Guardrails also need their own observability: every block should be logged with its reason, block rates should be monitored per category, and a sudden change in the rate is a signal worth alerting on, since it usually means either an attack or a regression in an upstream component.
| Check | Deterministic or Model-Based | Stage | Failure Response |
|---|---|---|---|
| Input length and rate limit | Deterministic | Before assembly | Reject with a clear error; cheap and unambiguous |
| PII redaction or tokenization | Deterministic pattern plus optional classifier | Before the call | Block the request; never send unredacted data on a check failure |
| Output schema conformance | Deterministic | After generation, before use | Retry once, then fail the request |
| Citation validity | Deterministic | After generation, before display | Strip the claim or refuse the answer |
| Tool call authorization and argument bounds | Deterministic | Before execution | Hard block; this is the action boundary |
| Safety and topic classification | Model-based | Input and output | Block and log; fail closed if the classifier is unavailable |
Worked Example: A Layered Guardrail Pipeline That Fails Closed
Note two properties of the code below. Deterministic checks run before any model-based classifier, because they are cheaper and more reliable. And every failure path, including the failure of a guardrail itself, results in a block rather than a pass.
async def guarded_answer(request, principal):
# ---- INPUT: prevent work rather than discard it ----
if count_tokens(request.text) > MAX_INPUT:
return Blocked('input_too_long')
await rate_limiter.acquire(principal.tenant_id)
try:
redacted, found = pii.redact(request.text) # deterministic patterns
except Exception:
return Blocked('redaction_unavailable') # FAIL CLOSED - never send
# unredacted data on error
if not await in_scope(redacted): # model-based, cheap model
return Blocked('out_of_scope')
# ---- GENERATE ----
out = await llm.complete(build_prompt(redacted), max_tokens=800, timeout=20)
# ---- OUTPUT: deterministic checks FIRST. Cheaper and more reliable. ----
parsed = schema.parse_or_none(out)
if parsed is None:
out = await llm.complete(repair_prompt(out), max_tokens=800)
parsed = schema.parse_or_none(out)
if parsed is None:
return Blocked('schema_violation')
if not set(parsed.citations) <= {c.id for c in request.chunks}:
return Blocked('invented_citation') # a fabricated source id
if urls_outside_allowlist(parsed.answer):
return Blocked('disallowed_url') # exfiltration vector
if pii.detect(parsed.answer):
return Blocked('pii_in_output')
# ---- Model-based safety LAST: the expensive, fuzzy layer ----
try:
verdict = await safety_classifier(parsed.answer, timeout=2.0)
except TimeoutError:
metrics.increment('guardrail.classifier_timeout')
return Blocked('safety_check_unavailable') # FAIL CLOSED, not open
if verdict.blocked:
return Blocked(verdict.category)
# ---- THE ACTION BOUNDARY: strictest gate in the system ----
if parsed.tool_call:
tc = parsed.tool_call
if tc.name not in principal.permitted_tools: # authorization, not
return Blocked('tool_not_permitted') # a suggestion
if not tool_schema[tc.name].validates(tc.args):
return Blocked('tool_args_invalid')
if tc.name in IRREVERSIBLE_TOOLS:
return RequiresConfirmation(tc) # human in the loop
log_guardrail_outcome(principal, blocked=False)
return parsedReal-World Examples
- The parcel delivery company DPD disabled the AI component of its customer support chatbot in January 2024 after a customer published an exchange in which the bot, prompted to do so, swore and composed a poem criticizing DPD as the worst delivery firm. The incident is a compact illustration that an output guardrail is a product requirement rather than a nicety, because the reputational cost of an unfiltered response is borne entirely by the company whose logo is on the widget.
- Purpose-built guardrail components became a standard category quickly, with Meta releasing Llama Guard in 2023 as a classifier model for screening both prompts and responses against a safety taxonomy, and NVIDIA releasing NeMo Guardrails as an open-source framework for defining programmable constraints around conversational applications. The emergence of dedicated small models for this job reflects that a cheap specialized classifier is a better fit than asking the main generation model to police itself.
Review Questions & Answers
Because they are cheaper, faster, completely reliable, and individually testable, and they catch a large share of real failures. Schema conformance, citation validity against the chunks actually supplied, URL allow-listing, length bounds, and pattern-based detection of regulated data are all objective properties of the output that require no judgment. Running them first means most bad outputs are rejected before any money or latency is spent on a classifier, and it also means the failures that do reach the classifier are the genuinely fuzzy ones it is actually needed for.
Because that is where an incorrect output stops being a bad answer and becomes a bad effect in the world. A hallucinated sentence shown to a user is a quality problem that a person can evaluate; a hallucinated tool call that deletes a record, sends an email, issues a refund, or executes code is an action taken on the system's authority with no human in between. Validation there must therefore be a hard gate rather than an advisory score: the tool must be on the caller's permitted list, the arguments must conform to the tool's schema and stay within permitted ranges, and irreversible operations should require explicit confirmation.
Once the first token has been written to the response, it cannot be retracted, so a guardrail that inspects the complete output runs after the user has already seen most of it. There are three workable options. Hold back a small sliding buffer so the guardrail scans a window of tokens before they are emitted, accepting a little added latency in exchange for the ability to block before display. Run lightweight incremental checks that can terminate the stream on a violation mid-generation. Or decline to stream content that genuinely must be fully validated first, such as a figure, a legal statement, or a tool call, and deliver it as a single complete event instead.
Because the block rate is a stable statistic under normal conditions, so a sharp movement in either direction indicates that something upstream changed. A spike usually means either an attack, such as a coordinated injection attempt, or a regression in a component feeding the model, such as a retrieval change that started supplying inappropriate context or a prompt release that altered output format. A collapse to near zero is often worse, because it typically means the guardrail itself stopped functioning, which is invisible in every other metric since blocked requests are not errors and unblocked ones look like ordinary successes.