Monitoring and Incident Response for LLM Apps
On this page
Why LLM apps need monitoring and incident response
Every control in this course lowers risk. None removes it. So the last layer is noticing quickly when something goes wrong, and being able to stop it. That needs three things in place before the incident: traces, alerts and kill switches.
Traces: record what the model saw and did
When an incident happens, the first question is "what was in the context when the bot did that?". You can only answer it if every turn is traced:
- the user and session, tenant and app version, including the prompt and model version,
- which documents and tool results entered the context, by id,
- every tool call with its arguments and the policy decision (Day 26),
- token usage and latency,
- the final output, after redaction of personal data (Day 9).
The OpenTelemetry GenAI semantic conventions give standard names for model calls and tool spans, and tools such as Langfuse, LangSmith or your existing tracing backend can store them. Record document ids, not full contents, and redact before storing: traces are a disclosure risk of their own.
Alerts that signal abuse
| Signal | What it may mean |
|---|---|
| Canary token in any output (Day 10) | Hidden context was extracted |
| Policy denials spike for one user or IP | Someone is probing your tools |
| Tool call to an unusual destination or amount | Injection or goal hijack in progress |
| Output filter hits for links, images or PII | Exfiltration attempt |
| Token use per user far above normal | Unbounded consumption, scraping, a loop |
| One document in many tainted sessions with denials | A poisoned record (Day 17) |
Start with a handful you can act on. An alert that fires every hour and gets ignored is worse than none.
Kill switches: turn off a capability in seconds
When an attack is live, you don't want a code deploy to stop it. Put each risky capability behind a runtime switch that the tool layer checks on every call:
import json, time
from pathlib import Path
FLAGS_FILE = Path("/etc/supportbot/flags.json") # in production: your feature-flag service
_cache = {"at": 0.0, "flags": {}}
def flags() -> dict:
if time.monotonic() - _cache["at"] > 5: # re-read every 5 seconds
try:
_cache["flags"] = json.loads(FLAGS_FILE.read_text())
except (OSError, ValueError):
_cache["flags"] = {"safe_mode": True} # unreadable flags: fail closed
_cache["at"] = time.monotonic()
return _cache["flags"]
def tool_enabled(name: str) -> bool:
f = flags()
if f.get("safe_mode"):
return name in {"lookup_order"} # read-only answers only
return f.get(f"tool.{name}", True)
def run_tool_guarded(name: str, args: dict, session) -> str:
if not tool_enabled(name):
audit("tool_disabled", tool=name, session=session.id)
return "That action isn't available right now. A support agent will follow up."
return run_tool(name, args, session)With {"tool.request_refund": false} an on-call engineer stops refunds within seconds. {"safe_mode": true} reduces SupportBot to read-only answers. Other useful switches: block one user or tenant, disable retrieval from one source, and pin the previous prompt and model version.
An incident response playbook for LLM apps
Write this down and rehearse it once, with SupportBot's F-03 finding from Day 25 as the drill scenario:
- Contain. Flip the switch for the affected capability. Block the attacker's account if known.
- Scope. Search traces for the pattern: which sessions, users and documents, and what data or actions were involved. Remove the poisoned record from the index.
- Eradicate. Fix the root cause in code (Days 8 to 19), not just the prompt.
- Recover. Turn the capability back on, watching the alerts.
- Learn. Add the attack to the CI gate (Day 24), update the threat model (Day 4), and meet notification duties if personal data was exposed.
Check yourself
Contain: turn off refunds with the kill switch, or put SupportBot in safe mode. Then pull kb-991 from the index and scope which refunds were issued from the traces.
Ids let you trace an incident back to its source without copying sensitive content into another system that's kept longer and seen by more people.
If the control plane is broken during an incident, failing closed keeps risky tools off. Failing open would quietly re-enable the capability you were trying to stop.
How to disable affected capabilities quickly, preserve traces as evidence, identify affected users and data, fix the root cause, and add a regression test before re-enabling.