Catalog My learning Blog
AI Security Course: OWASP LLM Top 10 2026 and Agentic AI · Phase 3 · Lesson 14 · 3 min read

Unexpected Code Execution in AI Agents (ASI05)

On this page
  1. How text becomes code
  2. Two strategies, depending on whether execution is the product
  3. A locked-down sandbox for generated Python
  4. Check yourself

How text becomes code

Some agents run code on purpose. Data-analysis assistants write and run Python, coding agents run tests and shell commands, and text-to-SQL tools execute the queries they generate. Others run code by accident: a developer passes model output to eval, exec, a shell, a template engine or a database without treating it as untrusted.

Either way, if an attacker can influence what the model writes, through direct or indirect injection, they can run code. OWASP lists this as ASI05, Unexpected Code Execution, and under the expanded LLM10, Improper Output Handling.

The accidental cases have a track record:

  • LangChain, 2023 (CVE-2023-29374). LLMMathChain passed model output to Python's exec, so a crafted question ran arbitrary code.
  • Vanna.AI, 2024 (CVE-2024-5565). The text-to-SQL library had the model write Plotly charting code and executed it. JFrog showed a prompt injection turning that into remote code execution.

In both, the bug was the same missing trust boundary: generated code was treated like code the developer wrote.

Two strategies, depending on whether execution is the product

PathExampleWhat to do
Code interpreter toolAgent runs generated PythonDisposable sandbox: no network, no secrets, limits
Accidental evalA library execs model outputRemove it and parse output as data
Generated SQLText-to-SQL runs queriesRead-only role, row-level security, or parameters only
Shell commandsAgent builds commands from textFixed commands with validated arguments
Generated templates and chartsPlotly or Jinja from the modelTreat as code: sandbox it or switch it off

When execution isn't the product, remove the path. Parse model output with strict schemas instead of evaluating it. Have the model choose values for parameterized queries rather than writing SQL. Never build shell commands by string concatenation. And check your dependencies for features that quietly execute generated code, like the charting helper in Vanna.

When execution is the product, isolate it. Assume the code is hostile, because sometimes it will be.

A locked-down sandbox for generated Python

Python
import pathlib, subprocess, tempfile, uuid

IMAGE = "python:3.12-slim"          # pin by digest in production

def run_untrusted_python(code: str, timeout_s: int = 20) -> dict:
    with tempfile.TemporaryDirectory() as work:
        pathlib.Path(work, "main.py").write_text(code)
        name = f"sandbox-{uuid.uuid4().hex[:12]}"
        cmd = [
            "docker", "run", "--rm", "--name", name,
            "--network", "none",                 # no exfiltration, no SSRF
            "--read-only",                       # immutable root file system
            "--tmpfs", "/tmp:rw,size=64m,noexec",
            "--volume", f"{work}:/work:ro",      # the code is mounted read-only
            "--workdir", "/work",
            "--user", "65534:65534",             # nobody
            "--cap-drop", "ALL",
            "--security-opt", "no-new-privileges",
            "--memory", "256m", "--cpus", "0.5", "--pids-limit", "64",
            "--env", "HOME=/tmp",                # no host environment, so no secrets
            IMAGE, "python", "-I", "main.py",
        ]
        try:
            p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout_s)
            return {"exit": p.returncode, "stdout": p.stdout[-10_000:], "stderr": p.stderr[-5_000:]}
        except subprocess.TimeoutExpired:
            subprocess.run(["docker", "kill", name], capture_output=True)
            return {"exit": -1, "stdout": "", "stderr": f"Timed out after {timeout_s}s"}

Each flag answers a specific attack. Code that tries to read secrets finds an empty environment. Code that calls an attacker's server has no network. A fork bomb hits the process limit, and an infinite loop hits the timeout. Every run starts from a clean image and is thrown away, so nothing carries over between users.

One limit to know: containers share the host kernel. For multi-tenant services running code from untrusted users, use stronger isolation such as gVisor or Firecracker micro-VMs, or a managed code-execution tool from your model provider, which runs in the provider's sandbox.

Check yourself

Stop executing model output. Have the model return a typed chart specification, such as chart type, columns and labels, validated against a schema, and render it with your own code.

Hijacked code would otherwise send data to an attacker or reach internal services. Removing the network closes both paths, whatever the code tries.

Run it with a read-only database role limited to the needed tables, with row-level security for the current user, plus a statement timeout. Even a hostile query then can't write, see other tenants' rows or run forever.

Isolation from the host, no network, a read-only filesystem except a scratch area, CPU, memory and time limits, a non-root user, and no access to secrets. Treat every run as untrusted.