Improper Output Handling in LLM Apps (LLM10)
On this page
Treat model output as untrusted input
Every classic injection has an LLM version. If model output reaches a browser unescaped, you get cross-site scripting. If it's pasted into SQL, you get SQL injection. Into a shell, command injection. The twist is that the attacker doesn't type into your form. They get the model to write the payload for them, through direct or indirect prompt injection.
LLM10, Improper Output Handling, dropped to tenth in 2026 but widened its scope. It now explicitly includes terminal control characters, the automatic fetching of external resources like images, and untrusted tool output flowing back through your application.
The rule is the one from the classic OWASP Top 10, applied to a new source: treat model output as untrusted input to the next component.
| Where output goes | Attack | Control |
|---|---|---|
| Browser (HTML or Markdown) | XSS, no-click data leaks via images | Escape HTML, strip images, allowlist links |
| SQL | SQL injection | Parameterized queries with model-chosen values only |
| Shell | Command injection | Fixed commands, validated arguments, no string building |
| Terminal or logs | ANSI escape sequences that rewrite or hide output | Strip control characters |
| Another system's API | Unexpected fields or values | Validate against a strict schema |
| Email and documents | Phishing links in trusted-looking messages | Link allowlist, plain text where possible |
Rendering SupportBot's answers safely
Day 7's attack relied on the widget rendering a Markdown image. Here's a renderer that escapes everything, removes images entirely, keeps only links to your own domains, and strips terminal control sequences:
import html, re
from urllib.parse import urlparse
ALLOWED_HOSTS = {"acme.example", "help.acme.example"}
ANSI = re.compile(r"\x1b\[[0-9;?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(\x07|\x1b\\)")
IMAGE = re.compile(r"!\[([^\]]*)\]\((?:[^()]|\([^()]*\))*\)")
LINK = re.compile(r"\[([^\]]+)\]\(((?:[^()\s]|\([^()\s]*\))+)\)")
def safe_link(m: re.Match) -> str:
text, url = m.group(1), m.group(2)
p = urlparse(url)
if p.scheme == "https" and p.hostname in ALLOWED_HOSTS:
return f'<a href="{html.escape(url, quote=True)}" rel="noopener nofollow">{html.escape(text)}</a>'
return html.escape(text) # keep the words, drop the link
def render_model_output(md: str) -> str:
md = ANSI.sub("", md) # terminal control sequences
md = IMAGE.sub(lambda m: html.escape(m.group(1)), md) # no remote images at all
parts, last = [], 0
for m in LINK.finditer(md):
parts.append(html.escape(md[last:m.start()]))
parts.append(safe_link(m))
last = m.end()
parts.append(html.escape(md[last:]))
return "".join(parts).replace("\n", "<br>")
evil = ('Done!  '
'<img src=x onerror=alert(1)> See [policy](https://help.acme.example/refunds) '
'or [here](javascript:alert(1)) or [this](https://attacker.example/x).\x1b[2J')
print(render_model_output(evil))The output:
Done! ok <img src=x onerror=alert(1)> See <a href="https://help.acme.example/refunds" rel="noopener nofollow">policy</a> or here or this.The image with the stolen email is gone, the raw HTML is shown as text, the javascript: and attacker links lose their targets, and the screen-clearing escape sequence is removed. Only the link to your help site survives.
In a real frontend, use your Markdown library's safe mode (raw HTML disabled), then apply the same image and link rules, and add a Content Security Policy that blocks images and connections to other origins. The CSP is the backstop if a rendering rule ever misses something. EchoLeak got past one by routing through a trusted Microsoft domain, which is why the allowlist above names exact hosts rather than whole providers.
SQL and shell: don't let the model write the command
If a feature lets the model choose a value, keep the query fixed and pass the value as a parameter:
# The model picks the status; the code owns the SQL.
status = tool_args["status"]
if status not in {"open", "shipped", "cancelled"}:
raise ValueError("bad status")
rows = db.execute("SELECT id, total FROM orders WHERE customer = ? AND status = ?",
(session.customer_id, status)).fetchall()The same goes for shells: call fixed programs with argument lists (subprocess.run(["git", "log", "-n", str(n)])), never shell=True with model text. When the model must write queries or code, that's Day 14's territory: a read-only role, or a sandbox.
Check yourself
Stored or reflected XSS from HTML the model was tricked into writing, plus no-click exfiltration through images. Disable raw HTML, strip images and allowlist links, with a CSP as a backstop.
A ticket can contain injected text that makes the model emit escape codes, which can hide lines, rewrite what's on screen or, in some terminals, trigger other behavior. Strip control characters before printing.
Big providers host user-controlled content and redirectors, and attackers use them to reach their own servers, as EchoLeak did. Exact hosts you control leave no such path.
Treat it like untrusted user input: encode it for the place it's used, validate it against a schema, and never pass it directly to a shell, SQL interpreter or HTML renderer.