# What is Red Teaming?

Also called Adversarial Testing, AI Red Teaming.

Red teaming is structured adversarial testing in which people deliberately try to make a system behave harmfully, leak data, or bypass its controls. Unlike a fixed evaluation suite, it is open-ended and creative, aiming to discover failure modes nobody anticipated. Findings are then converted into permanent test cases so the same weakness cannot silently return.

The distinction from evaluation is the direction of the question. An evaluation asks whether the system does what it should on known cases. Red teaming asks what the system can be made to do that it should not, with no fixed list. That open-endedness is the point, since anticipated failures are usually already covered and the costly incidents come from paths nobody modeled.

Scope for an AI system extends well past the model. It includes the prompt layer, the retrieval pipeline and everything that can be planted in it, tool definitions and their argument handling, permission boundaries between tenants, the approval flow and whether it can be skipped, and rate limits. A finding that a tool accepts an out-of-range identifier is as valuable as one about model output.

Teams mix manual and automated approaches. Human testers bring domain knowledge and lateral thinking, and specialists in a subject area find harms that generalists miss. Automated adversarial generation covers volume and can search phrasing space faster than people. Neither substitutes for the other, and involving people with different backgrounds materially changes which harms get found.

Red teaming appears in emerging governance expectations. The United States National Institute of Standards and Technology's AI Risk Management Framework references adversarial testing, and the European Union's AI Act places obligations on providers of general purpose models presenting systemic risk that include adversarial testing. Specific requirements depend on the framework, the system's classification, and the jurisdiction, and this area is still developing.

## Key points

- Open-ended adversarial testing, not a fixed pass or fail suite
- Scope covers prompts, retrieval, tools, permissions, and approvals
- Manual creativity and automated volume are complementary
- Diverse testers find harms that a uniform group misses
- Every finding becomes a permanent regression test

## In practice

Before enabling an assistant to send email, a team runs a two-week exercise. Testers plant instruction-shaped text in documents the assistant will read, attempt to reach another tenant's records through crafted references, and probe whether the approval step can be skipped by rephrasing a request. Three findings are fixed, and each becomes a permanent test that runs on every subsequent prompt or model change.

## Related terms

- [Model Evaluation](/en/glossary/model-evaluation)
- [Jailbreak](/en/glossary/jailbreak)
- [Prompt Injection](/en/glossary/prompt-injection)
- [AI Governance](/en/glossary/ai-governance)
- [Alignment](/en/glossary/alignment)

[Back to the AI Glossary](/en/glossary)