# What is Confidence Threshold?

Also called certainty cutoff.

A confidence threshold is a cutoff value that determines whether an agent acts on a result automatically or routes it for review. It requires a confidence signal that actually correlates with correctness, and a language model's own stated confidence is a weak signal, since models are frequently confident and wrong.

The concept is borrowed from classification systems, where a predicted probability is genuinely calibrated against observed accuracy. Language model agents rarely have such a signal by default. Asking a model to rate its confidence produces a number, but that number's relationship to correctness must be measured before any threshold based on it means anything.

More reliable signals come from structure rather than self-report. Agreement across multiple samples, whether a retrieved source actually supports a claim, whether a value passes validation or reconciles with a second system, and whether a required field was found rather than inferred are all observable facts. These can be combined into a score whose behavior can be measured.

A threshold is not chosen in the abstract, it is chosen against the cost of each error type. Where a wrong automatic action is expensive and a review is cheap, the threshold should sit high. Where review capacity is the constraint and errors are recoverable, it sits lower. Stating both costs makes the choice explicit rather than arbitrary.

Thresholds drift and must be revisited. A cutoff calibrated against one distribution of inputs will misbehave when the inputs change, usually by routing far more or far fewer items to review than intended. Tracking the review rate and the error rate among auto-accepted items is what makes the drift visible before it causes damage.

## Key points

- Cutoff deciding automatic action versus review
- Model self-reported confidence is poorly calibrated
- Structural signals like sample agreement are more reliable
- Set the cutoff from the cost of each error type
- Monitor review rate and post-acceptance errors for drift

## In practice

An invoice extraction agent scores each document by whether five samples agreed, whether the line items sum to the stated total, and whether the vendor matched an existing record. Documents scoring above the threshold post automatically; the rest go to a review queue. The queue holds about eight percent of volume, and any error found in the auto-posted remainder triggers a recalibration.

## Related terms

- [Self-Consistency](/en/glossary/self-consistency)
- [Human in the Loop](/en/glossary/human-in-the-loop)
- [Escalation Policy](/en/glossary/escalation-policy)
- [Fallback Behavior](/en/glossary/fallback-behavior)
- [Deterministic vs Probabilistic Behavior](/en/glossary/deterministic-vs-probabilistic-behavior)

[Back to the AI Glossary](/en/glossary)