Definition
AI red teaming is structured adversarial testing that searches for ways a system can fail, be misused, exploited, or pushed outside its constraints. Testers adopt plausible attacker or misuse goals, vary their access and techniques, and preserve the evidence needed to reproduce a successful finding.
The work should start with a threat model: what is in scope, which assets matter, what the tester can control, and what counts as success. Coverage may include model behavior, prompt injection, data leakage, tool misuse, permissions, supply-chain compromise, and failures that emerge only across a multi-step workflow.
A red-team exercise earns its value when findings become durable changes. Record the attack path, identify the failed boundary, add a regression test, fix the control, and verify that the system resists both the original exploit and reasonable variations.
Distinguish it from nearby terms
Ordinary evaluation measures performance on specified cases. Red teaming actively searches beyond expected use. Penetration testing focuses on exploitable technical weaknesses, while AI red teaming may also examine harmful behavior and misuse that do not fit a conventional software vulnerability.
Check your understanding
A team collects clever prompts but does not record model versions, tool permissions, or whether an attack actually caused an effect. It has anecdotes, not a reproducible red-team result.