Catalog My learning Blog
LLM System Design Course: RAG, Vector Databases, Cost · Phase 5 · Lesson 22 · 8 min read

LLM Guardrails: Input and Output Validation

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: A Layered Guardrail Pipeline That Fails Closed
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Design layered input and output controls around a model, decide where deterministic validation beats model-based classification, explain why guardrails must fail closed, and place validation correctly relative to streaming and to any action the output triggers.

Concept Explanation

Guardrails are the controls that sit between the model and everything else, and the useful way to think about them is as a specific application of a principle the security lessons of this course already established: an untrusted component's output must be validated before it is trusted, and an untrusted component's input must be constrained before it is accepted. What makes this harder than ordinary input validation is that the model is both a producer and a consumer of unbounded natural language, so neither side has a simple grammar to check against. What makes it tractable is that most of what needs checking is not actually about natural language at all.

That last point is the most practically valuable one in this lesson. Teams reach for a model-based classifier to check outputs when a deterministic rule would be faster, cheaper, more reliable, and testable. Does the response parse against the schema. Are all cited document identifiers present in the retrieved set. Does the answer contain a URL outside the allow-list. Does it contain something matching a credit card or national identifier pattern that should have been redacted. Is it within the length bound. Does the proposed tool call name a permitted tool with arguments of the right type and within permitted ranges. Every one of those is a deterministic check, and collectively they catch a large share of real incidents. Model-based classification should be reserved for the genuinely fuzzy judgments, principally safety categories and topic boundaries, where no rule can express the criterion.

Input controls come first and are cheaper than output controls because they prevent work rather than discard it. Bound the input length before assembly so an oversized document cannot inflate cost or push out retrieved context. Rate limit per authenticated principal, as established earlier. Redact or tokenize personal and regulated data before it leaves your boundary, which is a compliance control independent of any model behavior. Run a topical scope check when the product has a defined scope, so an off-topic request is declined cheaply rather than answered expensively. And apply injection-pattern screening as one layer among several, understanding from the next lesson that it is a filter rather than a solution.

Output controls need to be placed correctly relative to two things, and both placements are frequently wrong. The first is streaming: as the streaming lesson established, once the first token has been sent you cannot retract it, so a guardrail either operates on a sliding buffer held back from the user or the content is not streamed at all. The second and more important placement is relative to action. If the model output triggers anything - a tool call, a database write, an email, a payment, a code execution - validation must occur before that action, and it must be a hard gate rather than an advisory score. This is the boundary where a hallucination stops being a bad answer and becomes a bad action, and it is the single most important place in the entire system to be strict.

Finally, guardrails must fail closed, which connects directly to the exceptional-conditions material in this repository's security course. If the moderation service times out, if the validator throws, if the schema library errors, the safe response is to block or degrade rather than to pass the content through unchecked. The common anti-pattern is a try-catch around the guardrail that logs a warning and continues, which means the control silently disappears exactly when the system is under stress and least able to tolerate a bad output. Guardrails also need their own observability: every block should be logged with its reason, block rates should be monitored per category, and a sudden change in the rate is a signal worth alerting on, since it usually means either an attack or a regression in an upstream component.

CheckDeterministic or Model-BasedStageFailure Response
Input length and rate limitDeterministicBefore assemblyReject with a clear error; cheap and unambiguous
PII redaction or tokenizationDeterministic pattern plus optional classifierBefore the callBlock the request; never send unredacted data on a check failure
Output schema conformanceDeterministicAfter generation, before useRetry once, then fail the request
Citation validityDeterministicAfter generation, before displayStrip the claim or refuse the answer
Tool call authorization and argument boundsDeterministicBefore executionHard block; this is the action boundary
Safety and topic classificationModel-basedInput and outputBlock and log; fail closed if the classifier is unavailable

Worked Example: A Layered Guardrail Pipeline That Fails Closed

Note two properties of the code below. Deterministic checks run before any model-based classifier, because they are cheaper and more reliable. And every failure path, including the failure of a guardrail itself, results in a block rather than a pass.

Python
async def guarded_answer(request, principal):
    # ---- INPUT: prevent work rather than discard it ----
    if count_tokens(request.text) > MAX_INPUT:
        return Blocked('input_too_long')
    await rate_limiter.acquire(principal.tenant_id)

    try:
        redacted, found = pii.redact(request.text)      # deterministic patterns
    except Exception:
        return Blocked('redaction_unavailable')          # FAIL CLOSED - never send
                                                        # unredacted data on error
    if not await in_scope(redacted):                    # model-based, cheap model
        return Blocked('out_of_scope')

    # ---- GENERATE ----
    out = await llm.complete(build_prompt(redacted), max_tokens=800, timeout=20)

    # ---- OUTPUT: deterministic checks FIRST. Cheaper and more reliable. ----
    parsed = schema.parse_or_none(out)
    if parsed is None:
        out = await llm.complete(repair_prompt(out), max_tokens=800)
        parsed = schema.parse_or_none(out)
        if parsed is None:
            return Blocked('schema_violation')

    if not set(parsed.citations) <= {c.id for c in request.chunks}:
        return Blocked('invented_citation')              # a fabricated source id
    if urls_outside_allowlist(parsed.answer):
        return Blocked('disallowed_url')                 # exfiltration vector
    if pii.detect(parsed.answer):
        return Blocked('pii_in_output')

    # ---- Model-based safety LAST: the expensive, fuzzy layer ----
    try:
        verdict = await safety_classifier(parsed.answer, timeout=2.0)
    except TimeoutError:
        metrics.increment('guardrail.classifier_timeout')
        return Blocked('safety_check_unavailable')       # FAIL CLOSED, not open
    if verdict.blocked:
        return Blocked(verdict.category)

    # ---- THE ACTION BOUNDARY: strictest gate in the system ----
    if parsed.tool_call:
        tc = parsed.tool_call
        if tc.name not in principal.permitted_tools:     # authorization, not
            return Blocked('tool_not_permitted')         # a suggestion
        if not tool_schema[tc.name].validates(tc.args):
            return Blocked('tool_args_invalid')
        if tc.name in IRREVERSIBLE_TOOLS:
            return RequiresConfirmation(tc)              # human in the loop

    log_guardrail_outcome(principal, blocked=False)
    return parsed
Layered guardrails around a model call: input bounding, redaction and scope checks before generation, deterministic schema, citation, URL and PII checks after it, a model-based safety classifier last, and a hard authorization gate at the action boundary, with every failure path resolving to a block.
Layered guardrails around a model call: input bounding, redaction and scope checks before generation, deterministic schema, citation, URL and PII checks after it, a model-based safety classifier last, and a hard authorization gate at the action boundary, with every failure path resolving to a block.

Real-World Examples

  • The parcel delivery company DPD disabled the AI component of its customer support chatbot in January 2024 after a customer published an exchange in which the bot, prompted to do so, swore and composed a poem criticizing DPD as the worst delivery firm. The incident is a compact illustration that an output guardrail is a product requirement rather than a nicety, because the reputational cost of an unfiltered response is borne entirely by the company whose logo is on the widget.
  • Purpose-built guardrail components became a standard category quickly, with Meta releasing Llama Guard in 2023 as a classifier model for screening both prompts and responses against a safety taxonomy, and NVIDIA releasing NeMo Guardrails as an open-source framework for defining programmable constraints around conversational applications. The emergence of dedicated small models for this job reflects that a cheap specialized classifier is a better fit than asking the main generation model to police itself.
WarningA guardrail wrapped in a try-catch that logs a warning and continues is not a guardrail. Moderation timeouts, validator exceptions, and classifier outages occur precisely when the system is under stress, and a control that disappears under load provides no protection when it is most needed. Every guardrail failure path, including the failure of the guardrail itself, must block or degrade rather than pass content through unchecked.

Review Questions & Answers

Because they are cheaper, faster, completely reliable, and individually testable, and they catch a large share of real failures. Schema conformance, citation validity against the chunks actually supplied, URL allow-listing, length bounds, and pattern-based detection of regulated data are all objective properties of the output that require no judgment. Running them first means most bad outputs are rejected before any money or latency is spent on a classifier, and it also means the failures that do reach the classifier are the genuinely fuzzy ones it is actually needed for.

Because that is where an incorrect output stops being a bad answer and becomes a bad effect in the world. A hallucinated sentence shown to a user is a quality problem that a person can evaluate; a hallucinated tool call that deletes a record, sends an email, issues a refund, or executes code is an action taken on the system's authority with no human in between. Validation there must therefore be a hard gate rather than an advisory score: the tool must be on the caller's permitted list, the arguments must conform to the tool's schema and stay within permitted ranges, and irreversible operations should require explicit confirmation.

Once the first token has been written to the response, it cannot be retracted, so a guardrail that inspects the complete output runs after the user has already seen most of it. There are three workable options. Hold back a small sliding buffer so the guardrail scans a window of tokens before they are emitted, accepting a little added latency in exchange for the ability to block before display. Run lightweight incremental checks that can terminate the stream on a violation mid-generation. Or decline to stream content that genuinely must be fully validated first, such as a figure, a legal statement, or a tool call, and deliver it as a single complete event instead.

Because the block rate is a stable statistic under normal conditions, so a sharp movement in either direction indicates that something upstream changed. A spike usually means either an attack, such as a coordinated injection attempt, or a regression in a component feeding the model, such as a retrieval change that started supplying inappropriate context or a prompt release that altered output format. A collapse to near zero is often worse, because it typically means the guardrail itself stopped functioning, which is invisible in every other metric since blocked requests are not errors and unblocked ones look like ordinary successes.