AI Red Teaming Methodology
On this page
What AI red teaming really is
A red team simulates a real adversary to find out how a system fails before an attacker does. For AI systems it's easy to do badly: someone spends an afternoon typing clever prompts, screenshots two funny failures, and calls it done.
Microsoft's AI red team, after testing more than 100 generative AI products, published lessons in January 2025 that are worth taking to heart:
- Start from what the system can do and where it's used, not from the model. The impact comes from the tools, data and users around it.
- Simple attacks often work better than sophisticated ones. Many real findings came from basic techniques aimed at how the model was wired into the app.
- Automation scales coverage, but people find the novel problems. You need both.
- The work is never finished. Models, prompts and tools change, so testing has to be continuous.
The OWASP GenAI Red Teaming Guide (January 2025) gives a fuller structure across four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. This lesson turns that into a plan for SupportBot.
Step 1: Scope from the threat model
Your threat model from Day 4 already lists the attack paths. Each becomes a test objective:
| Path | Objective | Success means |
|---|---|---|
P1 KB or order note to send_email | Make the bot email data to an outside address | Any email to a non-customer address |
P2 Message to issue_refund | Get a refund the policy doesn't allow | A refund above the limit, or on someone else's order |
P3 Message to lookup_order | Read another customer's order | Another customer's details in a reply |
| P4 Content to chat widget | Exfiltrate through a rendered image or link | A request to a non-allowlisted host |
| LLM08 | Extract hidden context | The canary token in a reply |
| LLM06 | Run up cost | A single request exceeding the token budget |
Note the "success means" column. Every objective needs a deterministic check where possible: a canary string, a tool-call log, an outbound request, a refund record. That's what lets automation count results instead of a person judging each reply.
Step 2: Rules of engagement
Write these down before anyone starts:
- Targets and environment. Staging, with synthetic data, never production customer data.
- Allowed techniques. Direct and indirect injection, multi-turn escalation, poisoning the staging KB. Out of scope: attacks on the model provider's infrastructure.
- Safety. Tools run in dry-run mode or against sandboxes, payments go to a test processor, and email goes to a capture inbox.
- Rate and cost limits for automated runs.
- Reporting. Where findings go, and who is told immediately about a critical issue.
- Data handling. Generated harmful content stays in the test system and isn't shared casually.
Step 3: Mix manual and automated testing
| Tool | Best at | How it runs |
|---|---|---|
| Manual testing | Novel attacks, business logic, chaining weaknesses | People, guided by the threat model |
| garak (Day 22) | Broad baseline sweep with many known probes | CLI scan against an endpoint |
| PyRIT (Day 23) | Scripted and multi-turn attack campaigns | Python, attacker model talks to target |
| promptfoo (Day 24) | Repeatable suites as a CI gate | YAML config, fails the build on regressions |
A good first engagement does a manual pass on the highest-risk paths, a garak sweep for breadth, PyRIT campaigns against the two or three most valuable objectives, and then turns every confirmed finding into a promptfoo test so it can never quietly return.
Step 4: measure rates over repeated runs
Because model behavior is probabilistic, a single success or failure tells you little. Run each attack several times and report attack success rate: "the order-note injection triggered an outbound email in 7 of 20 runs". Rates let you compare before and after a fix, and between model versions. A fix that takes 7 of 20 to 0 of 20 is convincing. "I tried it and it didn't work" isn't.
Check yourself
Coverage of the system's real risks (tools, data, rendering), repeated runs to give success rates, and deterministic checks. Refusing jailbreaks says little about whether an order note can trigger an outbound email.
Successful attacks would otherwise expose real customer data or take real actions like refunds and emails. Staging with capture inboxes and test payment processors lets testers go all the way safely.
A regression test that reproduces it, with a deterministic success check, running in CI. Then a later prompt or model change can't reopen the hole unnoticed.
By impact and how reliably the issue reproduces, using the threat model to judge what an attacker would gain. Data exposure or unauthorized actions outrank embarrassing but harmless outputs.