Unexpected Code Execution in AI Agents (ASI05)
On this page
How text becomes code
Some agents run code on purpose. Data-analysis assistants write and run Python, coding agents run tests and shell commands, and text-to-SQL tools execute the queries they generate. Others run code by accident: a developer passes model output to eval, exec, a shell, a template engine or a database without treating it as untrusted.
Either way, if an attacker can influence what the model writes, through direct or indirect injection, they can run code. OWASP lists this as ASI05, Unexpected Code Execution, and under the expanded LLM10, Improper Output Handling.
The accidental cases have a track record:
- LangChain, 2023 (CVE-2023-29374).
LLMMathChainpassed model output to Python'sexec, so a crafted question ran arbitrary code. - Vanna.AI, 2024 (CVE-2024-5565). The text-to-SQL library had the model write Plotly charting code and executed it. JFrog showed a prompt injection turning that into remote code execution.
In both, the bug was the same missing trust boundary: generated code was treated like code the developer wrote.
Two strategies, depending on whether execution is the product
| Path | Example | What to do |
|---|---|---|
| Code interpreter tool | Agent runs generated Python | Disposable sandbox: no network, no secrets, limits |
| Accidental eval | A library execs model output | Remove it and parse output as data |
| Generated SQL | Text-to-SQL runs queries | Read-only role, row-level security, or parameters only |
| Shell commands | Agent builds commands from text | Fixed commands with validated arguments |
| Generated templates and charts | Plotly or Jinja from the model | Treat as code: sandbox it or switch it off |
When execution isn't the product, remove the path. Parse model output with strict schemas instead of evaluating it. Have the model choose values for parameterized queries rather than writing SQL. Never build shell commands by string concatenation. And check your dependencies for features that quietly execute generated code, like the charting helper in Vanna.
When execution is the product, isolate it. Assume the code is hostile, because sometimes it will be.
A locked-down sandbox for generated Python
import pathlib, subprocess, tempfile, uuid
IMAGE = "python:3.12-slim" # pin by digest in production
def run_untrusted_python(code: str, timeout_s: int = 20) -> dict:
with tempfile.TemporaryDirectory() as work:
pathlib.Path(work, "main.py").write_text(code)
name = f"sandbox-{uuid.uuid4().hex[:12]}"
cmd = [
"docker", "run", "--rm", "--name", name,
"--network", "none", # no exfiltration, no SSRF
"--read-only", # immutable root file system
"--tmpfs", "/tmp:rw,size=64m,noexec",
"--volume", f"{work}:/work:ro", # the code is mounted read-only
"--workdir", "/work",
"--user", "65534:65534", # nobody
"--cap-drop", "ALL",
"--security-opt", "no-new-privileges",
"--memory", "256m", "--cpus", "0.5", "--pids-limit", "64",
"--env", "HOME=/tmp", # no host environment, so no secrets
IMAGE, "python", "-I", "main.py",
]
try:
p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout_s)
return {"exit": p.returncode, "stdout": p.stdout[-10_000:], "stderr": p.stderr[-5_000:]}
except subprocess.TimeoutExpired:
subprocess.run(["docker", "kill", name], capture_output=True)
return {"exit": -1, "stdout": "", "stderr": f"Timed out after {timeout_s}s"}Each flag answers a specific attack. Code that tries to read secrets finds an empty environment. Code that calls an attacker's server has no network. A fork bomb hits the process limit, and an infinite loop hits the timeout. Every run starts from a clean image and is thrown away, so nothing carries over between users.
One limit to know: containers share the host kernel. For multi-tenant services running code from untrusted users, use stronger isolation such as gVisor or Firecracker micro-VMs, or a managed code-execution tool from your model provider, which runs in the provider's sandbox.
Check yourself
Stop executing model output. Have the model return a typed chart specification, such as chart type, columns and labels, validated against a schema, and render it with your own code.
Hijacked code would otherwise send data to an attacker or reach internal services. Removing the network closes both paths, whatever the code tries.
Run it with a read-only database role limited to the needed tables, with row-level security for the current user, plus a statement timeout. Even a hostile query then can't write, see other tenants' rows or run forever.
Isolation from the host, no network, a read-only filesystem except a scratch area, CPU, memory and time limits, a non-root user, and no access to secrets. Treat every run as untrusted.