What is Online Evaluation?
Also called online eval, production evaluation.
Online evaluation measures the quality of an AI system using real production traffic as it happens, rather than a stored test set. Signals come from user behavior, explicit ratings, downstream outcomes, and automated scoring applied to live outputs. It captures the real input distribution that a fixed dataset cannot represent.
Signals fall into three rough classes. Explicit feedback is a rating, a thumbs control, or a correction the user types. Implicit feedback is behavioral: whether the user retried, abandoned the session, edited the output, or accepted a suggestion unchanged. Outcome signals sit further downstream, such as whether a generated ticket resolution actually closed the ticket without reopening. Outcome signals are the most valuable and the hardest to attribute.
Automated scoring can also run on live traffic. A sampled fraction of production outputs is passed to programmatic checks or a model grader, producing a continuous quality metric alongside the usual latency and error metrics. Sampling keeps the added cost bounded, since scoring every output can approach the cost of generating it, and stratified sampling ensures rare but important request types are not lost.
The central difficulty is confounding. Traffic mix changes by hour and by season, a marketing campaign can flood the system with a new user type, and an upstream dependency can degrade quietly. A metric that moves is therefore ambiguous unless the change is isolated by a controlled comparison such as a split test or a shadow run. Online evaluation detects problems well and attributes causes poorly.
Privacy constraints shape what can be collected. Storing full production inputs and outputs for later scoring means storing user content, which brings retention limits, redaction requirements, and consent considerations. Many teams score in place and persist only the derived metric plus an identifier, which preserves the trend line without accumulating a large corpus of sensitive text.
Key points
- Uses live traffic instead of a stored test set
- Combines explicit ratings, behavioral signals, and downstream outcomes
- Sampled automated scoring keeps added cost bounded
- Detects problems well but attributes causes poorly
- Storing live inputs raises retention and privacy obligations
In practice
A drafting assistant logs, for every generated reply, whether the user sent it unchanged, edited it, or discarded it. A nightly job computes the unchanged-send rate by request type. After a prompt update, the rate for refund requests falls from 62 percent to 41 percent while other categories hold steady, prompting the team to inspect recent refund drafts and roll the prompt back.