Catalog My learning Blog
AI Security Course: OWASP LLM Top 10 2026 and Agentic AI · Phase 4 · Lesson 17 · 4 min read

Data and Model Poisoning in LLM Apps (LLM05)

On this page
  1. Small amounts of bad data go a long way
  2. Three routes in
  3. Controls against data and model poisoning
  4. Check yourself

Small amounts of bad data go a long way

It's tempting to think poisoning a model needs a large share of its training data. Two results say otherwise.

In October 2025, Anthropic, the UK AI Security Institute and the Alan Turing Institute reported that around 250 poisoned documents were enough to plant a backdoor in models from 600 million to 13 billion parameters. The number stayed roughly constant as the models and their datasets grew, so bigger models weren't safer.

For RAG, PoisonedRAG (Zou and colleagues, 2024) showed that injecting about five crafted texts per target question into a knowledge base of millions could make the system give the attacker's chosen answer roughly 90 percent of the time.

LLM05, Data and Model Poisoning, covers both, and the 2026 edition adds fine-tuning subversion and compromised adapters.

Three routes in

RouteHow it worksWho's exposed
Pre-training dataWeb pages crafted to be scraped into training setsModel builders
Fine-tuning dataBad examples in feedback, support logs or bought datasetsAnyone who fine-tunes
Retrieval corpusDocuments written to be retrieved for target questionsEvery RAG application

Pre-training is mostly the model provider's problem. Fine-tuning is yours if you fine-tune on customer conversations, thumbs-up feedback, or datasets you didn't build. A batch of crafted "good" examples can teach a model to approve refunds when a trigger phrase appears. Anthropic's Sleeper Agents research (2024) showed backdoors like that can survive standard safety training. Retrieval is the most practical attack against most apps, because anyone who can get text into your corpus can try it.

SupportBot's knowledge base, which many staff edit and which quotes customer text, is a retrieval-poisoning target. So is any "learn from resolved tickets" feature.

Controls against data and model poisoning

For fine-tuning data:

  • Know where every example came from, and keep that provenance with the dataset.
  • Never fine-tune directly on raw user input or feedback. Filter it, sample it and review it first.
  • Deduplicate and look for clusters of near-identical examples, a common sign of planted data.
  • Evaluate the tuned model against a held-out test set and a set of trigger probes before deploying it.

For retrieval corpora:

  • Control who can write to the corpus, and keep an audit trail of edits.
  • Separate trusted sources, such as reviewed policy articles, from untrusted ones, such as ticket text. Weight or label them differently in retrieval.
  • Review new or changed documents before they're indexed, at least for sensitive topics.
  • Watch for documents that suddenly rank top for many unrelated queries.

Here's a small version of that last check, which finds documents that dominate retrieval across your evaluation questions:

Python
from collections import Counter

def dominant_docs(retrieve, questions: list[str], k: int = 5, share: float = 0.5) -> list[tuple[str, float]]:
    """Flag documents that appear in the top k for an unusually large share of questions."""
    counts = Counter()
    for q in questions:
        counts.update(doc_id for doc_id, _score in retrieve(q, k=k))
    n = len(questions)
    return [(doc_id, c / n) for doc_id, c in counts.most_common() if c / n >= share]

# Example with a toy retriever: doc "kb-991" was crafted to match everything.
def toy_retrieve(q, k):
    return [("kb-991", 0.99), (f"kb-{q.split()[0]}", 0.5)][:k]

print(dominant_docs(toy_retrieve, ["refund window?", "shipping time?", "change address?"]))
# [('kb-991', 1.0)]

A help article about refunds should rank top for refund questions, not for shipping, passwords and returns at once. A document that does is worth a look.

Check yourself

They hold many conversations containing a trigger phrase and the behavior they want, such as approving refunds, and rate them five stars. Filter and review training examples, cap any single user's contribution, and test the tuned model with trigger probes before deploying.

It suggests the number of poisoned documents needed stays roughly constant as models and datasets grow, so a bigger training set doesn't dilute the attack.

Treat it as suspected retrieval poisoning. Check who added it and what it says, remove it from the index while reviewing, and add monitoring for documents that dominate retrieval.

Track the provenance of training and retrieval data, review what enters fine-tuning sets, restrict who can publish to knowledge bases, and evaluate models on held-out checks before deployment.