Agent observability connects a user request to every model call, tool call, subagent, MCP request, policy decision, and downstream service operation.
A final error message is not enough. Operators need the trajectory that produced it.
- Run envelope
agent.run carries trace_id 8f3… across the run.
- Model and retrieval
Record model, tokens, latency, and retrieval query, filters, and hits.
- Tool and network
Connect tool name, arguments, decision, MCP server and method, API status, retries, and effect ID.
- Outcome
Evaluate correctness, groundedness, policy compliance, task success, and human override alongside latency, call counts, retries, and cost.
A trace explains one run; quality and safety outcomes still need their own evidence.
How to read this model
A trace recorder with nested recording lanes. The object is a teaching analogy. Inspecting a part highlights its role; it does not execute or simulate the software. Cabinet trays are records, workpieces are data, and latches are checks. Unless an explanation states a quantity, compartments and sample workpieces do not denote data counts. Part dimensions are not a quantitative scale.
Trace one task across boundaries
Use a trace for the end-to-end task. Create spans for meaningful operations such as:
- Context assembly
- Model generation
- Retrieval
- Tool selection and execution
- Policy and approval decisions
- Subagent runs
- MCP calls
- Queue waits and background tasks
- Verification and compensation
Propagate W3C Trace Context through HTTP and supported protocol metadata. Keep stable task and operation identifiers in logs and audit records.
Record versions and outcomes
Useful attributes include model identifier, prompt version, tool name, tool-schema version, agent version, retry count, input and output tokens, cache use, latency, result status, and error class.
OpenTelemetry publishes GenAI semantic conventions, but several fields remain development-stage or have moved between packages. Pin the convention version used by instrumentation and plan migrations.
Prompts and tool arguments are sensitive
Prompts can contain personal data, proprietary documents, credentials, and attacker content. Tool arguments and results may be more sensitive than the final answer.
Default to metadata and structured redaction. Capture full content only under a documented policy with access controls, retention, and a reason. Sampling does not make secret logging safe.
Metrics answer different questions
Track service health with request rate, error rate, and duration. Track agent behavior with steps per task, tool calls, repeated actions, task completion, approvals, cancellations, and failure stage. Track economics with input tokens, output tokens, model cost, tool cost, and cost per successful task.
Use percentiles for latency. An average can hide a long tail caused by retries or one slow subagent.
Replay with recorded inputs and controlled effects
A replay should reproduce model and orchestration analysis without repeating external writes. Replace write tools with recorded results or a sandbox. Mark nondeterministic model output and compare trajectories rather than expecting byte-identical text.
Sources
Last verified: September 23, 2026.