Catalog My learning Blog
AI Security Course: OWASP LLM Top 10 2026 and Agentic AI · Phase 2 · Lesson 9 · 4 min read

Sensitive Information Disclosure in LLM Apps (LLM02)

On this page
  1. Whatever the model can see, someone can make it repeat
  2. Six ways LLM apps leak data, and the fix for each
  3. Fixing SupportBot's lookup and logs
  4. Check yourself

Whatever the model can see, someone can make it repeat

Sensitive information disclosure is second in the 2026 OWASP list because it's the payoff behind so many other attacks. Anything in the model's context can be echoed back. Anything you log can be read by an insider or exposed in a breach. Anything you send to a model provider leaves your boundary, under their terms.

Six ways LLM apps leak data, and the fix for each

LeakExampleControl
Over-broad toolslookup_order returns any order by idOwnership check in code, using the session identity
Stuffed contextThe full CRM profile in every prompt "just in case"Give the model only the fields the task needs
Shared stateA cache or memory keyed only by question textKey every cache and memory by tenant and user
Logs and tracesPrompts with personal data kept for a yearRedact before logging, short retention
MemorizationA fine-tuned model repeats training recordsKeep sensitive data out of training, test extraction
Users pastingConfidential code pasted into public chatbotsData classification and approved tools

Over-broad tools cause most real leaks. A lookup that trusts the id the model passes, or a search across all tenants, hands the model data the current user should never see. One plain question or one injection retrieves it. It's broken object-level authorization, the top API security risk, in an AI costume.

Memorization is real too. In 2021, Carlini and colleagues extracted hundreds of verbatim training sequences from GPT-2, including names, phone numbers and email addresses. Fine-tuning a model on customer data creates the same risk for your own model.

Users leak data as well. In April 2023 Samsung restricted staff use of ChatGPT after engineers pasted confidential source code and meeting notes into it. Whether a provider keeps or trains on inputs depends on the product and the contract, so data classification has to decide what may be sent at all.

Fixing SupportBot's lookup and logs

Day 3's lookup_order trusted the model's order id. The fix makes the session decide, returns only what an answer needs, and redacts personal data before anything is logged:

Python
import json, re

def lookup_order(session, order_id: str) -> str:
    order = ORDERS.get(order_id)
    # Authorization in code: the authenticated session decides, not the model's argument.
    if order is None or order["customer"] != session.verified_email:
        return json.dumps({"error": "No order with that id on your account."})
    # Data minimization: only what a support answer needs.
    return json.dumps({"id": order_id, "status": order["status"],
                       "total": order["total"], "eta": order["eta"]})

EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
PHONE = re.compile(r"\+?\d[\d\s().-]{7,}\d")
CARD = re.compile(r"\b(?:\d[ -]?){13,19}\b")

def redact(text: str) -> str:
    return PHONE.sub("[PHONE]", EMAIL.sub("[EMAIL]", CARD.sub("[CARD]", text)))

def log_turn(logger, session_id: str, user_msg: str, answer: str):
    logger.info("turn", extra={"session": session_id,
                               "user": redact(user_msg)[:2000],
                               "answer": redact(answer)[:2000]})

def output_guard(answer: str, session) -> str:
    # Last line of defense: no other customer's email may appear in a reply.
    for email in EMAIL.findall(answer):
        if email.lower() != session.verified_email.lower():
            return "Sorry, I can only share information about your own account."
    return answer

A few details matter:

  • The error is the same for "doesn't exist" and "not yours". That avoids confirming which order ids are real, the same rule as in web APIs.
  • The address isn't returned at all. A delivery-status answer doesn't need it, so it can't be leaked.
  • Regex redaction catches common formats. For names, addresses and other entities, libraries such as Microsoft Presidio go further.
  • The output guard is a last line, not the main defense. If it ever fires, something upstream is wrong, so log it and investigate.

Ask SupportBot about order A-200 as Ana now, and it says there's no such order on her account.

Check yourself

The model can be manipulated or simply mistaken. Access has to be enforced in the tool code by comparing the record's owner with the authenticated identity, whatever the model asks for.

One customer's cached answer, built from their private data, can be served to another customer who asks the same question. Key the cache by tenant and user, or only cache answers built from public data.

Redact personal data and secrets before writing traces, shorten retention to what debugging needs, and restrict access the same way you would for production data.

Anything in the context can end up in the output. Fetching only the current user's data, after an authorization check in code, means even a manipulated model has nothing extra to reveal.