What is Deterministic Replay?
Also called trace replay, run replay.
Deterministic replay is the reconstruction of a past agent run by replaying its recorded inputs, model outputs, and tool results in the original order, rather than re-executing the model and tools live. It makes a nondeterministic run reproducible for debugging, since the recorded outputs are fixed instead of being sampled again.
Re-running an agent does not reproduce it. Sampling is probabilistic, tool results change as underlying data changes, and timing differs, so the second run may never reach the step that failed. Replay sidesteps this by treating the recorded run as fixed data: the same prompts, the same returned completions, the same observations, in the same sequence.
This requires recording enough at the time. Prompts as actually sent including all injected context, raw model outputs, tool arguments and returned results, and the ordering and timing between them all have to be captured. Traces that store only a summary or the final answer cannot support replay, which is usually discovered only when a failure needs investigating.
Replay answers a specific question: what did the agent see and do. It cannot answer what the agent would do if something changed, because altering any input invalidates the recorded downstream outputs from that point on. Counterfactual work requires live re-execution against mocked tools, which is a related but different technique.
The main practical uses are debugging a specific reported failure, regression testing where a saved trace becomes a fixture, and reviewing agent decisions for compliance. Storage cost is the usual constraint, since full traces of long runs are large, so retention is commonly tiered by whether a run failed, escalated, or was flagged.
Key points
- Replays recorded outputs rather than re-sampling the model
- Requires full traces: prompts, outputs, tool arguments, results
- Answers what happened, not what would happen if changed
- Saved traces become fixtures for regression testing
- Storage cost drives tiered trace retention
In practice
A user reports that an agent quoted the wrong delivery window. Re-running the request returns a correct answer, because the underlying inventory data has since changed. Replaying the original trace shows the inventory tool returned an empty list and the agent inferred a window instead of saying it could not determine one, which identifies the real defect as a missing fallback.