What is Tracing?
Also called distributed tracing, request tracing.
Tracing records the path of a single request or task through a system as a tree of timed operations. Each trace carries an identifier that links every step, so a slow or failed run can be inspected end to end. In agent systems a trace typically covers model calls, tool calls, retries, and any handoffs between components.
Every request gets a unique trace identifier at its entry point. Each operation inside it opens a span that records its own start time, duration, and status, and links to the span that caused it. The identifier is propagated as work moves between services, queues, and background jobs, usually through request headers or message metadata, so the tree can be reassembled from data emitted by many processes.
Traces answer the question that metrics cannot: where a specific run actually spent its time and where it went wrong. For an agent that makes twenty calls in sequence, a total duration is not diagnostic, but a trace showing one retry loop and one oversized prompt is. Traces also carry attributes such as tenant and tool name, so the same data supports cost and usage attribution.
Two failure modes are common. The first is broken propagation: when a task moves onto a queue or into a new thread without carrying the trace context, the resulting trace splits into fragments that look unrelated. The second is sampling that keeps a fixed small share of traces, which discards precisely the rare slow runs an operator wants. Keeping all errors and slow runs is the usual correction.
A trace is made of spans, and a set of traces aggregated over time produces the metrics on a latency dashboard, so the three signals describe the same events at different resolutions. In agent systems, trajectory logging covers similar ground with a different emphasis, recording the semantic sequence of decisions and tool results rather than only the timing of operations.
Key points
- A trace is one request represented as a tree of timed spans.
- A shared trace identifier links steps across services and queues.
- Reveals where time and money were actually spent in a run.
- Context must be propagated across async boundaries or traces fragment.
In practice
A user reports that a research task took two minutes. The trace shows eleven spans: one planning model call of four seconds, eight web fetches of about two seconds each, one retry after a timeout, and a final summarization call of ninety seconds. The summarization step, not the web fetching, is the bottleneck, so the team shortens the input passed into it.