What is Quality Assurance Review?
Also called QA review, output review.
A quality assurance review is a structured evaluation of completed work against a defined standard, performed after the work is delivered rather than before. Applied to AI agents, it means sampling finished output, scoring it on a rubric, and feeding the findings back into procedures. It differs from approval, which happens before delivery.
The distinction from approval is practical. Approval gates every item and adds latency to all of it. Review samples a fraction after the fact, costs far less, and is the only workable option once volume is high. Most mature deployments run both, with approval reserved for high consequence categories and review covering everything else.
Rubrics need to be specific enough that two reviewers scoring the same item agree. Useful dimensions include factual accuracy against a verifiable source, completeness against the request, correct tone for the audience, and adherence to the documented procedure. Vague criteria such as quality produce inconsistent scores and findings that cannot be acted on.
Sampling design determines what the review can detect. Purely random sampling is unbiased but may never surface a rare, serious failure. Stratified sampling, which deliberately over-samples unusual cases, new procedures, and high value items, finds the failures that matter at the cost of a representative overall rate. Many teams run both and report them separately.
The output has to close a loop or the exercise is decorative. A finding should terminate in a changed procedure, a clarified instruction, an added exception rule, or a narrowed scope, with a record of what changed and when. Review programs that only produce scores tend to be abandoned, correctly, once the novelty passes.
Key points
- Samples completed work against a rubric after delivery
- Differs from approval, which gates before delivery
- Rubric must be specific enough for reviewer agreement
- Stratified sampling finds rare failures random sampling misses
- Findings must change a procedure or the review is decorative
In practice
A firm reviews twenty agent-drafted client emails each week, ten drawn at random and ten from accounts flagged as sensitive, scoring each on accuracy, completeness, and tone. Three consecutive weeks show the same omission of the next-step date. The procedure is amended to require an explicit next step in every email, and the following month's reviews confirm the omission stops.