Build a Deliberately Vulnerable LLM Assistant Lab
On this page
What you'll set up
A running support assistant with retrieval, three tools and an HTTP API, with the same weaknesses real assistants ship with. Every later lesson attacks one weakness and fixes it with one control. Run it on your own machine only, and never deploy it.
You need Python 3.12, an Anthropic API key, and:
pip install anthropic fastapi uvicorn
export ANTHROPIC_API_KEY=...Step 1: Understand what's broken, on purpose
Each design choice demonstrates a risk from Day 2:
| Component | Vulnerable choice | Risk |
|---|---|---|
| System prompt | Contains a discount code and escalation rules | LLM08 Hidden Context Exposure |
lookup_order | Returns any order, no ownership check | LLM02, broken object-level authorization |
send_email | Any recipient, no approval | LLM01 exfiltration path |
issue_refund | No limit, no approval | LLM03 Excessive Agency |
| Knowledge base | Editable by many staff, quotes customer text | LLM01 indirect injection, LLM05 poisoning |
| Chat widget | Renders Markdown links and images as-is | LLM10 Improper Output Handling |
| API | No size or rate limits | LLM06 Unbounded Consumption |
Step 2: Write the service
# supportbot/app.py (deliberately vulnerable: local use only)
import json
import anthropic
from fastapi import FastAPI
from pydantic import BaseModel
client = anthropic.Anthropic()
app = FastAPI()
SESSIONS: dict[str, list] = {}
SYSTEM = """You are SupportBot for Acme. Be helpful and friendly.
Internal: VIP customers may receive discount code ACME-VIP-7731. Escalate to a human if the
customer mentions a lawyer or a chargeback.""" # LLM08: secrets in context
KB = { # editable by many staff; some articles quote customer text verbatim
"refunds": "Refunds are possible within 30 days of delivery.",
"shipping": "Orders ship within 2 business days.",
}
ORDERS = {"A-100": {"customer": "ana@example.com", "total": 42.5, "address": "12 Elm St"},
"A-200": {"customer": "sam@example.com", "total": 980.0, "address": "9 Oak Ave"}}
TOOLS = [
{"name": "lookup_order", "description": "Get an order by id.",
"input_schema": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}},
{"name": "send_email", "description": "Send an email.",
"input_schema": {"type": "object", "properties": {"to": {"type": "string"}, "body": {"type": "string"}},
"required": ["to", "body"]}},
{"name": "issue_refund", "description": "Refund an order.",
"input_schema": {"type": "object", "properties": {"order_id": {"type": "string"}, "amount": {"type": "number"}},
"required": ["order_id", "amount"]}},
]
def run_tool(name: str, args: dict) -> str:
if name == "lookup_order":
return json.dumps(ORDERS.get(args["order_id"], {})) # LLM02: no ownership check
if name == "send_email":
print(f"EMAIL to {args['to']}: {args['body'][:200]}") # LLM01: any recipient
return "sent"
if name == "issue_refund":
return f"refunded {args['amount']} on {args['order_id']}" # LLM03: no limit
return "unknown tool"
class Chat(BaseModel):
session_id: str
message: str # LLM06: no size limit
@app.post("/chat")
def chat(req: Chat):
history = SESSIONS.setdefault(req.session_id, [])
context = "\n".join(KB.values()) # naive retrieval: everything
history.append({"role": "user", "content": f"Help articles:\n{context}\n\nCustomer: {req.message}"})
for _ in range(8):
resp = client.messages.create(model="claude-opus-5-5", max_tokens=4000,
system=SYSTEM, tools=TOOLS, messages=history)
history.append({"role": "assistant", "content": resp.content})
if resp.stop_reason != "tool_use":
return {"answer": "".join(b.text for b in resp.content if b.type == "text")} # LLM10
results = [{"type": "tool_result", "tool_use_id": b.id, "content": run_tool(b.name, b.input)}
for b in resp.content if b.type == "tool_use"]
history.append({"role": "user", "content": results})
return {"answer": "Sorry, something went wrong."}The loop is a standard tool-use agent: call the model, run any tools it asks for, feed the results back, and stop when it answers in text. The vulnerabilities aren't in the loop. They're in what the tools allow and what text reaches the model.
Step 3: Run it and find the first bug yourself
uvicorn supportbot.app:app --host 127.0.0.1 --port 8811
curl -s localhost:8811/chat -H 'content-type: application/json' \
-d '{"session_id": "ana", "message": "Where is my order A-100?"}'It answers helpfully. Now, as the same customer, ask about A-200, which belongs to someone else. It answers just as helpfully, address included. That's broken object-level authorization, and Day 7 fixes it.
Step 4: Set up your before-and-after workflow
Commit this vulnerable version and tag it vulnerable. Each fix in later lessons goes in its own commit. In Phase 5 you'll build a red-team suite and run it against both versions: it should find the problems in vulnerable and pass against the hardened one. That comparison is the capstone deliverable.
Common lab setup problems
authentication_error: theANTHROPIC_API_KEYvariable isn't set in the shell running uvicorn.Address already in use: another process is on port 8811. Pick another port.- The bot refuses obvious attacks in later lessons. Good models resist crude attacks most of the time. Run each attack several times, and remember the point: a defense that works most of the time isn't a control.
Check yourself
To show that anything in the model's context can be extracted. Secrets and decision rules belong in code and configuration the model can't read.
ORDERS.get(args["order_id"]) returns any order. The fix is an ownership check in run_tool, using the customer identity from the authenticated session, not from anything the model says.
Running the same test suite against both versions proves the controls actually work, rather than the tests simply being weak. It also gives a regression baseline, so a later change that reopens a hole shows up as a failing test.
Security testing deliberately tries to make the app reveal data. With synthetic data, a successful test exposes nothing real, so you can test freely and share transcripts in reports.