Catalog My learning Blog
Build AI Agents with MCP: 30-Day Course · Phase 6 · Lesson 29 · 5 min read

AI Agent Guardrails and Human-in-the-Loop Approval

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: A Tool Policy Engine with Deny-First Rules
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Design layered controls for an agent: input and output guardrails, tiered permissions for tools, approval workflows that people actually read, sandboxing, budgets and kill switches, and audit logs, so that the agent can be autonomous where it is safe and supervised where it is not.

Concept Explanation

A guardrail is any control that constrains what an agent can do or say. The guiding principle is defense in depth: no single layer is reliable, so you combine several, and the most important ones are enforced by code outside the model. A system prompt that says "never delete data" is guidance; a database role without delete permission is a guardrail.

Start with permissions, because they bound the worst case. Classify every tool by its effect: read-only, reversible write, irreversible or external effect, and financially or legally significant. Then decide a policy per class. Reads can usually run freely. Reversible writes may run automatically within limits, such as refunds under a threshold or edits on a branch. Irreversible actions, external communication and anything above a threshold need human approval. Some actions should never be available to the agent at all. Encode the policy as rules in the host, not as prompt text, and apply deny rules before allow rules.

Approvals only help if people read them. Show the exact action and arguments in plain language, with the context that matters: "Send email to 4,312 customers with subject X", not "Allow tool call?". Batch low-risk approvals, and keep high-risk ones rare enough that people do not click through them reflexively. Approval fatigue is a real failure mode, which is why coding agents now offer classifier-assisted modes that approve routine actions automatically and stop only for risky ones.

Input and output guardrails catch problems at the edges. Input checks can detect obvious injection attempts, off-topic requests or personal data that should not be processed. Output checks can block secrets, verify that responses cite retrieved sources, and enforce formats with schemas. These checks are probabilistic when they use models, so treat them as additional layers, never as the only defense.

Containment limits damage when everything else fails. Run code and shell tools in sandboxes or containers without production credentials. Give agents separate, narrowly scoped identities rather than a human's full access. Enforce budgets on steps, tokens, money and time. Provide a kill switch that stops all running agents. And keep an audit log of every action with its approver, so incidents can be investigated and responsibility is clear.

Responsibility does not transfer to the model. In February 2024 a Canadian tribunal held Air Canada liable for incorrect refund advice its website chatbot gave a customer, rejecting the argument that the chatbot was a separate entity. In July 2025 an AI coding agent on Replit deleted a company's production database during a code freeze, despite instructions not to make changes; the platform responded by separating development and production databases. Both cases point to the same lesson: an organization owns what its agents do, so the controls must be real.

LayerExampleEnforced by
Permissions by tool classReads free, refunds over 100 EUR need approval, no deletesHost policy engine
Least-privilege identityRead-only database role, repo-scoped tokenCredentials and infrastructure
Human approvalClear description of action and argumentsHost UI
Input and output checksInjection screening, secret scanning, schema validationCode and classifiers
ContainmentSandbox, budgets, kill switchRuntime
AccountabilityAudit log with approver and argumentsLogging pipeline

Worked Example: A Tool Policy Engine with Deny-First Rules

The host calls decide before every tool execution. Rules are data, deny rules win, thresholds are explicit, and every decision is written to an audit log.

Code
import fnmatch, json, time
from typing import Literal

Decision = Literal["allow", "ask", "deny"]

POLICY = {
    "deny":  ["*__delete_*", "billing__change_plan", "email__send_bulk"],
    "ask":   ["email__send", "github__merge_pull_request", "deploy__*"],
    "allow": ["*__search_*", "*__get_*", "*__list_*", "notes__create_note"],
    "limits": {"billing__issue_refund": {"field": "amount_eur", "auto_max": 100}},
}

def decide(tool: str, args: dict) -> tuple[Decision, str]:
    if any(fnmatch.fnmatch(tool, p) for p in POLICY["deny"]):
        return "deny", "Tool is not permitted for this agent."
    limit = POLICY["limits"].get(tool)
    if limit:
        value = float(args.get(limit["field"], 0))
        if value <= limit["auto_max"]:
            return "allow", f"{limit['field']} {value} is within the automatic limit."
        return "ask", f"{limit['field']} {value} exceeds the automatic limit of {limit['auto_max']}."
    if any(fnmatch.fnmatch(tool, p) for p in POLICY["ask"]):
        return "ask", "Tool requires human approval."
    if any(fnmatch.fnmatch(tool, p) for p in POLICY["allow"]):
        return "allow", "Matched allow rule."
    return "ask", "No rule matched; defaulting to approval."      # unknown tools are never auto-run

def audit(tool: str, args: dict, decision: str, reason: str, approver: str | None = None):
    with open("agent-audit.jsonl", "a") as f:
        f.write(json.dumps({"ts": time.time(), "tool": tool, "args": args, "decision": decision,
                            "reason": reason, "approver": approver}) + "\n")

print(decide("billing__issue_refund", {"amount_eur": 40}))     # ('allow', ...)
print(decide("billing__issue_refund", {"amount_eur": 400}))    # ('ask', ...)
print(decide("crm__delete_contact", {"id": 7}))                # ('deny', ...)

The default for unknown tools is ask, not allow, so a newly connected MCP server cannot quietly gain autonomy. When the decision is deny, the host returns an error result to the model explaining that the action is not permitted, so the agent can tell the user instead of retrying.

Controls are layered and the important ones live outside the model: deny-first tool policy with thresholds, least-privilege identities, clear human approvals, input and output checks, containment with budgets and a kill switch, and an audit log.
Controls are layered and the important ones live outside the model: deny-first tool policy with thresholds, least-privilege identities, clear human approvals, input and output checks, containment with budgets and a kill switch, and an audit log.

Real-World Examples

  • In Moffatt v. Air Canada (February 2024), British Columbia's Civil Resolution Tribunal ordered Air Canada to honor refund terms its chatbot had invented, ruling that the company was responsible for all information on its website, including the chatbot's answers.
  • In July 2025 Replit's AI agent deleted the production database of SaaStr founder Jason Lemkin's project during a declared code freeze; Replit's CEO apologized and the company introduced automatic separation of development and production databases.
Key TakeawaysLayer an agent's guardrails, and enforce the important ones in code rather than relying on instructions in the prompt. Classify tools by effect, apply deny-first policies with thresholds, give agents least-privilege identities, and require clear, rare approvals for irreversible or external actions. Add input and output checks as extra layers, contain execution with sandboxes, budgets and kill switches, and log every action. The organization remains responsible for what its agents do.

Review Questions & Answers

The model may not follow it, especially under prompt injection or unusual circumstances. A guardrail is enforced outside the model, for example by permissions, credentials or a host policy that blocks the action.

It should default to requiring approval or denial, never automatic execution, so newly added tools do not silently gain autonomy.

It describes the exact action and arguments in plain language with relevant context, appears only for consequential actions, and is rare enough that people read it rather than approving by reflex.

The organization is responsible for what its agents say and do; it cannot disclaim liability by treating the agent as a separate entity, so controls must prevent harmful actions and statements.