Catalog My learning Blog
AI Security Course: OWASP LLM Top 10 2026 and Agentic AI · Phase 6 · Lesson 27 · 4 min read

Monitoring and Incident Response for LLM Apps

On this page
  1. Why LLM apps need monitoring and incident response
  2. Traces: record what the model saw and did
  3. Alerts that signal abuse
  4. Kill switches: turn off a capability in seconds
  5. An incident response playbook for LLM apps
  6. Check yourself

Why LLM apps need monitoring and incident response

Every control in this course lowers risk. None removes it. So the last layer is noticing quickly when something goes wrong, and being able to stop it. That needs three things in place before the incident: traces, alerts and kill switches.

Traces: record what the model saw and did

When an incident happens, the first question is "what was in the context when the bot did that?". You can only answer it if every turn is traced:

  • the user and session, tenant and app version, including the prompt and model version,
  • which documents and tool results entered the context, by id,
  • every tool call with its arguments and the policy decision (Day 26),
  • token usage and latency,
  • the final output, after redaction of personal data (Day 9).

The OpenTelemetry GenAI semantic conventions give standard names for model calls and tool spans, and tools such as Langfuse, LangSmith or your existing tracing backend can store them. Record document ids, not full contents, and redact before storing: traces are a disclosure risk of their own.

Alerts that signal abuse

SignalWhat it may mean
Canary token in any output (Day 10)Hidden context was extracted
Policy denials spike for one user or IPSomeone is probing your tools
Tool call to an unusual destination or amountInjection or goal hijack in progress
Output filter hits for links, images or PIIExfiltration attempt
Token use per user far above normalUnbounded consumption, scraping, a loop
One document in many tainted sessions with denialsA poisoned record (Day 17)

Start with a handful you can act on. An alert that fires every hour and gets ignored is worse than none.

Kill switches: turn off a capability in seconds

When an attack is live, you don't want a code deploy to stop it. Put each risky capability behind a runtime switch that the tool layer checks on every call:

Python
import json, time
from pathlib import Path

FLAGS_FILE = Path("/etc/supportbot/flags.json")     # in production: your feature-flag service
_cache = {"at": 0.0, "flags": {}}

def flags() -> dict:
    if time.monotonic() - _cache["at"] > 5:          # re-read every 5 seconds
        try:
            _cache["flags"] = json.loads(FLAGS_FILE.read_text())
        except (OSError, ValueError):
            _cache["flags"] = {"safe_mode": True}    # unreadable flags: fail closed
        _cache["at"] = time.monotonic()
    return _cache["flags"]

def tool_enabled(name: str) -> bool:
    f = flags()
    if f.get("safe_mode"):
        return name in {"lookup_order"}              # read-only answers only
    return f.get(f"tool.{name}", True)

def run_tool_guarded(name: str, args: dict, session) -> str:
    if not tool_enabled(name):
        audit("tool_disabled", tool=name, session=session.id)
        return "That action isn't available right now. A support agent will follow up."
    return run_tool(name, args, session)

With {"tool.request_refund": false} an on-call engineer stops refunds within seconds. {"safe_mode": true} reduces SupportBot to read-only answers. Other useful switches: block one user or tenant, disable retrieval from one source, and pin the previous prompt and model version.

An incident response playbook for LLM apps

Write this down and rehearse it once, with SupportBot's F-03 finding from Day 25 as the drill scenario:

  1. Contain. Flip the switch for the affected capability. Block the attacker's account if known.
  2. Scope. Search traces for the pattern: which sessions, users and documents, and what data or actions were involved. Remove the poisoned record from the index.
  3. Eradicate. Fix the root cause in code (Days 8 to 19), not just the prompt.
  4. Recover. Turn the capability back on, watching the alerts.
  5. Learn. Add the attack to the CI gate (Day 24), update the threat model (Day 4), and meet notification duties if personal data was exposed.

Check yourself

Contain: turn off refunds with the kill switch, or put SupportBot in safe mode. Then pull kb-991 from the index and scope which refunds were issued from the traces.

Ids let you trace an incident back to its source without copying sensitive content into another system that's kept longer and seen by more people.

If the control plane is broken during an incident, failing closed keeps risky tools off. Failing open would quietly re-enable the capability you were trying to stop.

How to disable affected capabilities quickly, preserve traces as evidence, identify affected users and data, fix the root cause, and add a regression test before re-enabling.