---
title: 'AI red teaming'
description: 'Structured adversarial testing intended to discover ways an AI system can fail, be misused, or violate constraints.'
canonical_url: 'https://darkfactory.dev/glossary/red-teaming'
markdown_url: 'https://darkfactory.dev/glossary/red-teaming.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# AI red teaming


## Definition

AI red teaming is structured adversarial testing that searches for ways a system can fail, be misused, exploited, or pushed outside its constraints. Testers adopt plausible attacker or misuse goals, vary their access and techniques, and preserve the evidence needed to reproduce a successful finding.

The work should start with a threat model: what is in scope, which assets matter, what the tester can control, and what counts as success. Coverage may include model behavior, prompt injection, data leakage, tool misuse, permissions, supply-chain compromise, and failures that emerge only across a multi-step workflow.

A red-team exercise earns its value when findings become durable changes. Record the attack path, identify the failed boundary, add a regression test, fix the control, and verify that the system resists both the original exploit and reasonable variations.

## Distinguish it from nearby terms

Ordinary evaluation measures performance on specified cases. Red teaming actively searches beyond expected use. Penetration testing focuses on exploitable technical weaknesses, while AI red teaming may also examine harmful behavior and misuse that do not fit a conventional software vulnerability.

## Check your understanding

A team collects clever prompts but does not record model versions, tool permissions, or whether an attack actually caused an effect. It has anecdotes, not a reproducible red-team result.

## Related terms

- [Adversarial example](https://darkfactory.dev/glossary/adversarial-example)
- [Prompt injection](https://darkfactory.dev/glossary/prompt-injection)

## Evidence and further reading

- [NIST AI 100-2: Adversarial Machine Learning](https://csrc.nist.gov/pubs/ai/100/2/e2025/final)
- [OWASP GenAI Security Glossary](https://genai.owasp.org/glossary/)