What is Evaluation Harness?
Also called eval harness, eval framework.
An evaluation harness is the software scaffolding that runs a set of test inputs through an AI system, collects the outputs, scores them against defined criteria, and reports aggregate results. It standardizes how quality is measured so that two runs, two prompts, or two model configurations can be compared on identical terms rather than on impressions.
A harness usually has four separable parts: a dataset loader that supplies inputs and any reference answers, a runner that executes the system under test with controlled settings, one or more scorers that turn each output into a number or label, and a reporter that aggregates and stores results. Keeping these parts separate lets teams swap the scorer without touching the dataset, or point the same dataset at a different system.
Scorers vary widely in rigor. Exact match and numeric tolerance work for closed-form answers. String similarity and overlap metrics work for extraction. Programmatic checks, such as whether generated code compiles or whether a tool call has valid arguments, are often the most reliable signal for agents. Model-graded scoring covers open-ended text where no reference answer exists, at the cost of introducing a second system whose own errors must be characterized.
For agents, the unit under evaluation is contested. Some harnesses score only the final answer, which is simple but blind to a run that reached the right result through unsafe or wasteful steps. Others score the full trajectory, checking whether the expected tools were called in a defensible order. Trajectory scoring detects more failure modes but is brittle, because many valid paths exist for the same task.
Harnesses are most useful when they run on a schedule or on every change, with results kept in a durable store so trends are visible over time. A harness that runs only during a crisis produces a single number with nothing to compare it against. Recording the exact prompt, configuration, dataset revision, and code version alongside each result is what makes a later comparison meaningful.
Key points
- Separates dataset, runner, scorer, and reporter into swappable parts
- Programmatic checks are usually more reliable than text similarity
- Agent harnesses may score final output, trajectory, or both
- Results need stored configuration and versions to be comparable
- Value comes from repeated runs, not one-off measurement
In practice
A team builds a harness for a support agent. The loader reads 300 archived tickets with labeled correct resolutions. The runner replays each ticket against the agent with tool calls pointed at a sandbox. Scorers check three things: whether the resolution label matches, whether any write tool was called, and whether the reply cites a real help article. The reporter writes one row per ticket plus a summary, keyed by commit.