Capstone: Ship a Production AI Agent with MCP
On this page
Learning Objective
Bring the course together by designing a production support-operations agent end to end: scope, architecture, MCP servers, context and memory, guardrails, evaluation, observability and a staged rollout. Produce a design you could defend in a review.
Concept Explanation
The capstone scenario is common and demanding: a support-operations agent for a software company. It reads incoming tickets, investigates using internal systems, drafts replies, issues small credits, and escalates anything risky to a human. It will touch customer data, money and external communication, so every topic in this course applies.
Start with scope and shape (Days 1 and 6). Not every ticket needs an agent. A router classifies tickets: password resets and known issues go to deterministic workflows with templates; billing disputes, bug reports and complex questions go to the agent. Define done precisely: a draft reply grounded in data, an action taken within policy, or an escalation with a summary. Define what the agent must never do: change plans, delete data, or send messages without approval in the first phase.
Next, tools as MCP servers (Days 11 to 22). The agent needs a ticketing server, a billing server, a logs server and a knowledge base server. Prefer vendor servers where they exist; build thin internal servers for the rest with task-shaped tools, read-only annotations where true, structured output, and elicitation for confirmations. Remote servers run statelessly behind your gateway, authenticate with OAuth using audience-bound tokens, and use a service identity with least privilege. The host namespaces tools, pins definitions, and caches lists.
Then context and memory (Days 5 and 9). The system prompt states the role, the policy summary and the definition of done. Each ticket starts a fresh session. Stable knowledge such as product facts lives in the knowledge base, retrieved just in time. Long-term memory is limited to account notes written with provenance and reviewed by staff.
Security and guardrails (Days 23 and 29) come next. Tickets are untrusted content, the billing system is private data, and email is external communication: the full lethal trifecta. Break it deliberately. The agent drafts replies but a human sends them in phase one, and credits above a small threshold require approval. A deny-first policy engine sits in the host, and every action is audited.
Finally, quality and operations (Days 27 and 28). Build an evaluation set from two hundred historical tickets with known resolutions, grade outcomes with code where possible and a calibrated rubric judge for reply quality, and measure pass^k. Trace every ticket, track cost per resolved ticket and cache hit ratio, and alert on loops and error spikes. Roll out in stages: shadow mode where the agent drafts but nothing is used, then assisted mode with human review of every draft, then autonomy for narrow categories that have proven reliable, always with a quick path back to humans.
| Phase | Agent may | Gate to next phase |
|---|---|---|
| Shadow | Investigate and draft; nothing is shown to customers | Drafts match human resolutions on the eval set and in sampled review |
| Assisted | Drafts shown to agents for editing and sending; credits require approval | High acceptance rate, no policy violations for several weeks |
| Narrow autonomy | Send replies for low-risk categories; credits under threshold | Stable pass^k, low complaint rate, kill switch tested |
| Expanded | Add categories one at a time | Per-category evaluations and monitoring |
Worked Example: The Agent's Design as Reviewable Configuration
Writing the design down as configuration makes it reviewable and testable. Each section maps to a lesson in this course.
from dataclasses import dataclass, field
@dataclass
class AgentDesign:
name: str = "support-ops-agent"
model: str = "claude-opus-5-5"
router_model: str = "claude-haiku-4-5"
workflow_routes: dict = field(default_factory=lambda: { # Day 6: not everything is an agent
"password_reset": "template:reset", "known_outage": "template:status-page"})
mcp_servers: dict = field(default_factory=lambda: { # Days 11-22
"tickets": {"url": "https://mcp.helpdesk.example.com/mcp", "auth": "oauth", "scopes": ["tickets:rw"]},
"billing": {"url": "https://billing.internal/mcp", "auth": "oauth", "scopes": ["invoices:read", "credits:write"]},
"logs": {"url": "https://logs.internal/mcp", "auth": "oauth", "scopes": ["logs:read"]},
"kb": {"url": "https://kb.internal/mcp", "auth": "oauth", "scopes": ["kb:read"]}})
policy: dict = field(default_factory=lambda: { # Days 23, 29
"deny": ["billing__change_plan", "*__delete_*"],
"ask": ["tickets__send_reply"], # phase 1: humans send
"limits": {"billing__issue_credit": {"field": "amount_eur", "auto_max": 25}}})
limits: dict = field(default_factory=lambda: {"max_steps": 25, "max_usd": 0.75, "timeout_s": 180})
context: dict = field(default_factory=lambda: { # Days 5, 9
"clear_tool_results_after": 6, "memory": "account-notes, reviewed, with provenance"})
evals: dict = field(default_factory=lambda: { # Day 27
"dataset": "evals/historical_tickets_200.jsonl", "k": 5,
"gates": {"pass^5": 0.80, "policy_violations": 0}})
observability: dict = field(default_factory=lambda: { # Day 28
"otel_endpoint": "https://otel.internal:4317", "alert_on": ["steps>20", "cost_usd>0.75", "error_rate>5%"]})
rollout_phase: str = "shadow" # Day 30 staged rollout
DESIGN = AgentDesign()In review, each field invites a question: why this threshold, which tickets fall in the eval set, who receives the alerts, what triggers a rollback. A design that can answer them is ready to ship to shadow mode.
Real-World Examples
- Klarna reported in February 2024 that its AI assistant handled two thirds of customer service chats in its first month, and later in 2025 said it was again investing in human agents so customers could always reach a person, illustrating why staged autonomy and an easy path to humans matter.
- Intercom, Zendesk and Salesforce all launched AI support agents in 2024 and 2025 with configurable policies, approval steps and handoff to humans, the same layered design this capstone uses.
Review Questions & Answers
Simple, well-understood cases such as password resets are cheaper, faster and more reliable as deterministic workflows. The agent's flexibility is reserved for cases that need investigation.
It removes autonomous external communication: the agent drafts replies but humans send them, and credits above a small threshold require approval, so injected instructions in tickets cannot directly exfiltrate data or move money.
The agent investigates and drafts on real tickets but nothing it produces reaches customers. It lets the team measure quality and safety on real traffic with zero customer risk before granting any autonomy.
A stable, high pass^k on the evaluation set for the category, a high acceptance rate of drafts by human reviewers, zero policy violations over a sustained period, and a tested kill switch and rollback path.