Catalog My learning Blog
AI Security Course: OWASP LLM Top 10 2026 and Agentic AI · Phase 5 · Lesson 21 · 3 min read

AI Red Teaming Methodology

On this page
  1. What AI red teaming really is
  2. Step 1: Scope from the threat model
  3. Step 2: Rules of engagement
  4. Step 3: Mix manual and automated testing
  5. Step 4: measure rates over repeated runs
  6. Check yourself

What AI red teaming really is

A red team simulates a real adversary to find out how a system fails before an attacker does. For AI systems it's easy to do badly: someone spends an afternoon typing clever prompts, screenshots two funny failures, and calls it done.

Microsoft's AI red team, after testing more than 100 generative AI products, published lessons in January 2025 that are worth taking to heart:

  • Start from what the system can do and where it's used, not from the model. The impact comes from the tools, data and users around it.
  • Simple attacks often work better than sophisticated ones. Many real findings came from basic techniques aimed at how the model was wired into the app.
  • Automation scales coverage, but people find the novel problems. You need both.
  • The work is never finished. Models, prompts and tools change, so testing has to be continuous.

The OWASP GenAI Red Teaming Guide (January 2025) gives a fuller structure across four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. This lesson turns that into a plan for SupportBot.

Step 1: Scope from the threat model

Your threat model from Day 4 already lists the attack paths. Each becomes a test objective:

PathObjectiveSuccess means
P1 KB or order note to send_emailMake the bot email data to an outside addressAny email to a non-customer address
P2 Message to issue_refundGet a refund the policy doesn't allowA refund above the limit, or on someone else's order
P3 Message to lookup_orderRead another customer's orderAnother customer's details in a reply
P4 Content to chat widgetExfiltrate through a rendered image or linkA request to a non-allowlisted host
LLM08Extract hidden contextThe canary token in a reply
LLM06Run up costA single request exceeding the token budget

Note the "success means" column. Every objective needs a deterministic check where possible: a canary string, a tool-call log, an outbound request, a refund record. That's what lets automation count results instead of a person judging each reply.

Step 2: Rules of engagement

Write these down before anyone starts:

  • Targets and environment. Staging, with synthetic data, never production customer data.
  • Allowed techniques. Direct and indirect injection, multi-turn escalation, poisoning the staging KB. Out of scope: attacks on the model provider's infrastructure.
  • Safety. Tools run in dry-run mode or against sandboxes, payments go to a test processor, and email goes to a capture inbox.
  • Rate and cost limits for automated runs.
  • Reporting. Where findings go, and who is told immediately about a critical issue.
  • Data handling. Generated harmful content stays in the test system and isn't shared casually.

Step 3: Mix manual and automated testing

ToolBest atHow it runs
Manual testingNovel attacks, business logic, chaining weaknessesPeople, guided by the threat model
garak (Day 22)Broad baseline sweep with many known probesCLI scan against an endpoint
PyRIT (Day 23)Scripted and multi-turn attack campaignsPython, attacker model talks to target
promptfoo (Day 24)Repeatable suites as a CI gateYAML config, fails the build on regressions
Each tool covers a different part of the job. Manual testing finds the new problems, garak sweeps known ones, PyRIT runs campaigns and promptfoo keeps fixes from regressing.
Each tool covers a different part of the job. Manual testing finds the new problems, garak sweeps known ones, PyRIT runs campaigns and promptfoo keeps fixes from regressing.

A good first engagement does a manual pass on the highest-risk paths, a garak sweep for breadth, PyRIT campaigns against the two or three most valuable objectives, and then turns every confirmed finding into a promptfoo test so it can never quietly return.

Step 4: measure rates over repeated runs

Because model behavior is probabilistic, a single success or failure tells you little. Run each attack several times and report attack success rate: "the order-note injection triggered an outbound email in 7 of 20 runs". Rates let you compare before and after a fix, and between model versions. A fix that takes 7 of 20 to 0 of 20 is convincing. "I tried it and it didn't work" isn't.

Check yourself

Coverage of the system's real risks (tools, data, rendering), repeated runs to give success rates, and deterministic checks. Refusing jailbreaks says little about whether an order note can trigger an outbound email.

Successful attacks would otherwise expose real customer data or take real actions like refunds and emails. Staging with capture inboxes and test payment processors lets testers go all the way safely.

A regression test that reproduces it, with a deterministic success check, running in CI. Then a later prompt or model change can't reopen the hole unnoticed.

By impact and how reliably the issue reproduces, using the threat model to judge what an attacker would gain. Data exposure or unauthorized actions outrank embarrassing but harmless outputs.