Catalog My learning Blog
Build AI Agents with MCP: 30-Day Course · Phase 6 · Lesson 30 · 5 min read

Capstone: Ship a Production AI Agent with MCP

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: The Agent's Design as Reviewable Configuration
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Bring the course together by designing a production support-operations agent end to end: scope, architecture, MCP servers, context and memory, guardrails, evaluation, observability and a staged rollout. Produce a design you could defend in a review.

Concept Explanation

The capstone scenario is common and demanding: a support-operations agent for a software company. It reads incoming tickets, investigates using internal systems, drafts replies, issues small credits, and escalates anything risky to a human. It will touch customer data, money and external communication, so every topic in this course applies.

Start with scope and shape (Days 1 and 6). Not every ticket needs an agent. A router classifies tickets: password resets and known issues go to deterministic workflows with templates; billing disputes, bug reports and complex questions go to the agent. Define done precisely: a draft reply grounded in data, an action taken within policy, or an escalation with a summary. Define what the agent must never do: change plans, delete data, or send messages without approval in the first phase.

Next, tools as MCP servers (Days 11 to 22). The agent needs a ticketing server, a billing server, a logs server and a knowledge base server. Prefer vendor servers where they exist; build thin internal servers for the rest with task-shaped tools, read-only annotations where true, structured output, and elicitation for confirmations. Remote servers run statelessly behind your gateway, authenticate with OAuth using audience-bound tokens, and use a service identity with least privilege. The host namespaces tools, pins definitions, and caches lists.

Then context and memory (Days 5 and 9). The system prompt states the role, the policy summary and the definition of done. Each ticket starts a fresh session. Stable knowledge such as product facts lives in the knowledge base, retrieved just in time. Long-term memory is limited to account notes written with provenance and reviewed by staff.

Security and guardrails (Days 23 and 29) come next. Tickets are untrusted content, the billing system is private data, and email is external communication: the full lethal trifecta. Break it deliberately. The agent drafts replies but a human sends them in phase one, and credits above a small threshold require approval. A deny-first policy engine sits in the host, and every action is audited.

Finally, quality and operations (Days 27 and 28). Build an evaluation set from two hundred historical tickets with known resolutions, grade outcomes with code where possible and a calibrated rubric judge for reply quality, and measure pass^k. Trace every ticket, track cost per resolved ticket and cache hit ratio, and alert on loops and error spikes. Roll out in stages: shadow mode where the agent drafts but nothing is used, then assisted mode with human review of every draft, then autonomy for narrow categories that have proven reliable, always with a quick path back to humans.

PhaseAgent mayGate to next phase
ShadowInvestigate and draft; nothing is shown to customersDrafts match human resolutions on the eval set and in sampled review
AssistedDrafts shown to agents for editing and sending; credits require approvalHigh acceptance rate, no policy violations for several weeks
Narrow autonomySend replies for low-risk categories; credits under thresholdStable pass^k, low complaint rate, kill switch tested
ExpandedAdd categories one at a timePer-category evaluations and monitoring

Worked Example: The Agent's Design as Reviewable Configuration

Writing the design down as configuration makes it reviewable and testable. Each section maps to a lesson in this course.

Python
from dataclasses import dataclass, field

@dataclass
class AgentDesign:
    name: str = "support-ops-agent"
    model: str = "claude-opus-5-5"
    router_model: str = "claude-haiku-4-5"
    workflow_routes: dict = field(default_factory=lambda: {          # Day 6: not everything is an agent
        "password_reset": "template:reset", "known_outage": "template:status-page"})
    mcp_servers: dict = field(default_factory=lambda: {              # Days 11-22
        "tickets": {"url": "https://mcp.helpdesk.example.com/mcp", "auth": "oauth", "scopes": ["tickets:rw"]},
        "billing": {"url": "https://billing.internal/mcp", "auth": "oauth", "scopes": ["invoices:read", "credits:write"]},
        "logs":    {"url": "https://logs.internal/mcp", "auth": "oauth", "scopes": ["logs:read"]},
        "kb":      {"url": "https://kb.internal/mcp", "auth": "oauth", "scopes": ["kb:read"]}})
    policy: dict = field(default_factory=lambda: {                   # Days 23, 29
        "deny": ["billing__change_plan", "*__delete_*"],
        "ask": ["tickets__send_reply"],                              # phase 1: humans send
        "limits": {"billing__issue_credit": {"field": "amount_eur", "auto_max": 25}}})
    limits: dict = field(default_factory=lambda: {"max_steps": 25, "max_usd": 0.75, "timeout_s": 180})
    context: dict = field(default_factory=lambda: {                  # Days 5, 9
        "clear_tool_results_after": 6, "memory": "account-notes, reviewed, with provenance"})
    evals: dict = field(default_factory=lambda: {                    # Day 27
        "dataset": "evals/historical_tickets_200.jsonl", "k": 5,
        "gates": {"pass^5": 0.80, "policy_violations": 0}})
    observability: dict = field(default_factory=lambda: {            # Day 28
        "otel_endpoint": "https://otel.internal:4317", "alert_on": ["steps>20", "cost_usd>0.75", "error_rate>5%"]})
    rollout_phase: str = "shadow"                                    # Day 30 staged rollout

DESIGN = AgentDesign()

In review, each field invites a question: why this threshold, which tickets fall in the eval set, who receives the alerts, what triggers a rollback. A design that can answer them is ready to ship to shadow mode.

The agent starts in shadow mode, drafting without customer impact, moves to assisted mode where humans edit and send, then gains narrow autonomy for proven low-risk categories, expanding one category at a time.
The agent starts in shadow mode, drafting without customer impact, moves to assisted mode where humans edit and send, then gains narrow autonomy for proven low-risk categories, expanding one category at a time.

Real-World Examples

  • Klarna reported in February 2024 that its AI assistant handled two thirds of customer service chats in its first month, and later in 2025 said it was again investing in human agents so customers could always reach a person, illustrating why staged autonomy and an easy path to humans matter.
  • Intercom, Zendesk and Salesforce all launched AI support agents in 2024 and 2025 with configurable policies, approval steps and handoff to humans, the same layered design this capstone uses.
Key TakeawaysA production AI agent is a whole system, not just a prompt: tools, memory, guardrails, evaluation and monitoring all have to work together. Route simple cases to workflows, expose capabilities through well-designed and authorized MCP servers, manage context and memory deliberately, break the lethal trifecta with policy and approvals, and prove quality with evaluations before granting autonomy. Instrument everything, roll out in stages from shadow to narrow autonomy, and keep a fast path back to humans.

Review Questions & Answers

Simple, well-understood cases such as password resets are cheaper, faster and more reliable as deterministic workflows. The agent's flexibility is reserved for cases that need investigation.

It removes autonomous external communication: the agent drafts replies but humans send them, and credits above a small threshold require approval, so injected instructions in tickets cannot directly exfiltrate data or move money.

The agent investigates and drafts on real tickets but nothing it produces reaches customers. It lets the team measure quality and safety on real traffic with zero customer risk before granting any autonomy.

A stable, high pass^k on the evaluation set for the category, a high acceptance rate of drafts by human reviewers, zero policy violations over a sustained period, and a tested kill switch and rollback path.