# What is Safety Evaluation?

Also called Safety Eval, Safety Testing.

Safety evaluation is the systematic testing of an AI system's behavior against defined harm categories, measuring how often it produces disallowed content, complies with manipulation attempts, or takes unsafe actions. It complements capability evaluation, which measures what a system can do, and it usually combines automated test suites with adversarial human probing.

A safety evaluation begins with an explicit policy, because a measurement is only meaningful relative to a stated line. The policy defines harm categories, what constitutes a violation, and what an acceptable response looks like, including partial compliance and safe alternatives. Vague policies produce evaluations where graders disagree and scores move for reasons unrelated to the system.

Test sets typically pair harmful prompts with benign prompts that superficially resemble them. This dual structure is essential because a system can trivially score perfectly on the harmful set by refusing everything. Reporting violation rate alongside over-refusal rate on the benign set is what makes the number interpretable, and both should be tracked across releases.

Grading is done by rubric based human review, automated classifiers, or a model acting as judge, each with tradeoffs in cost, throughput and reliability. Judge models require their own validation against human labels, since agreement varies by category and can shift when either the judged model or the judge changes. Confidence intervals matter, as small test sets produce noisy differences.

The main limitation is that static evaluations measure known risks under known phrasings. Real adversaries adapt, and published benchmarks eventually leak into training data, inflating scores without improving behavior. Safety evaluation is therefore treated as one input alongside red teaming, production monitoring and incident review, not as a certificate that a system is safe.

## Key points

- Measures behavior against a written harm policy
- Pair harmful prompts with lookalike benign prompts
- Report violation rate and over-refusal rate together
- Model graders need validation against human labels
- Static benchmarks decay as they leak into training data

## In practice

Before a prompt change ships, a suite runs several hundred cases spanning harm categories the product's policy names, plus a matched set of benign requests that use similar vocabulary. Results are graded by a rubric and spot checked by a reviewer. The change reduces violations in one category but raises refusals on benign medical questions, so it is revised rather than released on the improved headline number.

## Related terms

- [Model Evaluation](/en/glossary/model-evaluation)
- [Red Teaming](/en/glossary/red-teaming)
- [Over-Refusal](/en/glossary/over-refusal)
- [Output Safety](/en/glossary/output-safety)
- [AI Guardrails](/en/glossary/ai-guardrails)

[Back to the AI Glossary](/en/glossary)