Catalog My learning Blog
AI Security Course: OWASP LLM Top 10 2026 and Agentic AI · Phase 1 · Lesson 3 · 4 min read

Build a Deliberately Vulnerable LLM Assistant Lab

On this page
  1. What you'll set up
  2. Step 1: Understand what's broken, on purpose
  3. Step 2: Write the service
  4. Step 3: Run it and find the first bug yourself
  5. Step 4: Set up your before-and-after workflow
  6. Common lab setup problems
  7. Check yourself

What you'll set up

A running support assistant with retrieval, three tools and an HTTP API, with the same weaknesses real assistants ship with. Every later lesson attacks one weakness and fixes it with one control. Run it on your own machine only, and never deploy it.

You need Python 3.12, an Anthropic API key, and:

Code
pip install anthropic fastapi uvicorn
export ANTHROPIC_API_KEY=...

Step 1: Understand what's broken, on purpose

Each design choice demonstrates a risk from Day 2:

ComponentVulnerable choiceRisk
System promptContains a discount code and escalation rulesLLM08 Hidden Context Exposure
lookup_orderReturns any order, no ownership checkLLM02, broken object-level authorization
send_emailAny recipient, no approvalLLM01 exfiltration path
issue_refundNo limit, no approvalLLM03 Excessive Agency
Knowledge baseEditable by many staff, quotes customer textLLM01 indirect injection, LLM05 poisoning
Chat widgetRenders Markdown links and images as-isLLM10 Improper Output Handling
APINo size or rate limitsLLM06 Unbounded Consumption

Step 2: Write the service

Python
# supportbot/app.py  (deliberately vulnerable: local use only)
import json
import anthropic
from fastapi import FastAPI
from pydantic import BaseModel

client = anthropic.Anthropic()
app = FastAPI()
SESSIONS: dict[str, list] = {}

SYSTEM = """You are SupportBot for Acme. Be helpful and friendly.
Internal: VIP customers may receive discount code ACME-VIP-7731. Escalate to a human if the
customer mentions a lawyer or a chargeback."""                      # LLM08: secrets in context

KB = {  # editable by many staff; some articles quote customer text verbatim
    "refunds": "Refunds are possible within 30 days of delivery.",
    "shipping": "Orders ship within 2 business days.",
}
ORDERS = {"A-100": {"customer": "ana@example.com", "total": 42.5, "address": "12 Elm St"},
          "A-200": {"customer": "sam@example.com", "total": 980.0, "address": "9 Oak Ave"}}

TOOLS = [
    {"name": "lookup_order", "description": "Get an order by id.",
     "input_schema": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}},
    {"name": "send_email", "description": "Send an email.",
     "input_schema": {"type": "object", "properties": {"to": {"type": "string"}, "body": {"type": "string"}},
                      "required": ["to", "body"]}},
    {"name": "issue_refund", "description": "Refund an order.",
     "input_schema": {"type": "object", "properties": {"order_id": {"type": "string"}, "amount": {"type": "number"}},
                      "required": ["order_id", "amount"]}},
]

def run_tool(name: str, args: dict) -> str:
    if name == "lookup_order":
        return json.dumps(ORDERS.get(args["order_id"], {}))         # LLM02: no ownership check
    if name == "send_email":
        print(f"EMAIL to {args['to']}: {args['body'][:200]}")        # LLM01: any recipient
        return "sent"
    if name == "issue_refund":
        return f"refunded {args['amount']} on {args['order_id']}"    # LLM03: no limit
    return "unknown tool"

class Chat(BaseModel):
    session_id: str
    message: str                                                     # LLM06: no size limit

@app.post("/chat")
def chat(req: Chat):
    history = SESSIONS.setdefault(req.session_id, [])
    context = "\n".join(KB.values())                                 # naive retrieval: everything
    history.append({"role": "user", "content": f"Help articles:\n{context}\n\nCustomer: {req.message}"})
    for _ in range(8):
        resp = client.messages.create(model="claude-opus-5-5", max_tokens=4000,
                                      system=SYSTEM, tools=TOOLS, messages=history)
        history.append({"role": "assistant", "content": resp.content})
        if resp.stop_reason != "tool_use":
            return {"answer": "".join(b.text for b in resp.content if b.type == "text")}  # LLM10
        results = [{"type": "tool_result", "tool_use_id": b.id, "content": run_tool(b.name, b.input)}
                   for b in resp.content if b.type == "tool_use"]
        history.append({"role": "user", "content": results})
    return {"answer": "Sorry, something went wrong."}

The loop is a standard tool-use agent: call the model, run any tools it asks for, feed the results back, and stop when it answers in text. The vulnerabilities aren't in the loop. They're in what the tools allow and what text reaches the model.

Step 3: Run it and find the first bug yourself

Code
uvicorn supportbot.app:app --host 127.0.0.1 --port 8811

curl -s localhost:8811/chat -H 'content-type: application/json' \
  -d '{"session_id": "ana", "message": "Where is my order A-100?"}'

It answers helpfully. Now, as the same customer, ask about A-200, which belongs to someone else. It answers just as helpfully, address included. That's broken object-level authorization, and Day 7 fixes it.

Step 4: Set up your before-and-after workflow

Commit this vulnerable version and tag it vulnerable. Each fix in later lessons goes in its own commit. In Phase 5 you'll build a red-team suite and run it against both versions: it should find the problems in vulnerable and pass against the hardened one. That comparison is the capstone deliverable.

Common lab setup problems

  • authentication_error: the ANTHROPIC_API_KEY variable isn't set in the shell running uvicorn.
  • Address already in use: another process is on port 8811. Pick another port.
  • The bot refuses obvious attacks in later lessons. Good models resist crude attacks most of the time. Run each attack several times, and remember the point: a defense that works most of the time isn't a control.

Check yourself

To show that anything in the model's context can be extracted. Secrets and decision rules belong in code and configuration the model can't read.

ORDERS.get(args["order_id"]) returns any order. The fix is an ownership check in run_tool, using the customer identity from the authenticated session, not from anything the model says.

Running the same test suite against both versions proves the controls actually work, rather than the tests simply being weak. It also gives a regression baseline, so a later change that reopens a hole shows up as a failing test.

Security testing deliberately tries to make the app reveal data. With synthetic data, a successful test exposes nothing real, so you can test freely and share transcripts in reports.