The Paradigm Shift: Deterministic → Probabilistic
Traditional software fails loudly. An unhandled exception throws a stack trace. A bad database query returns an error code. You know exactly what broke and where.
AI agents fail silently. The same prompt submitted twice can take completely different execution paths. A retrieval step returns garbage context. The model hallucinates. Your monitoring dashboard stays green the whole time.
Traditional: Explicit Failures
Exceptions, stack traces, HTTP 5xx errors. Every failure mode is loud and localized. Monitoring looks for known signatures.
Agents: Silent Failures
Wrong answers returned at 200 OK. Hallucinations, bad context, looping tool calls — all invisible to traditional metrics.
What You Need Instead
Runtime state reconstruction, causal trace linking, context quality verification, and probabilistic evaluation.
| Feature | Deterministic Monitoring | Probabilistic Observability |
|---|---|---|
| Code Execution | Static, pre-compiled logic; fixed control flows | Dynamic, model-generated logic; non-deterministic tool selection |
| Failure Modes | Explicit exceptions, stack traces, HTTP 5xx | Silent failures (200 OK), hallucinations, low-relevance retrieval |
| State Complexity | Bounded local memory, predictable request scope | Dynamic: multi-turn history, vector embeddings, transient scratchpads |
| Root Cause | Stack trace inspection, local reproduction | Causal trace tree reconstruction, context audit, token entropy analysis |
Railway Simulator: See the Difference
Think of a static commuter railway vs. a dynamic agent network. The commuter train stops loudly when a rail breaks. The agent network silently routes to the wrong city — and every sensor along the route reports perfect health.
The static railway fails loud and early. The agent network continues at full speed and only reveals its error when you check the final destination — which, in production, no one does automatically.
Compounding Error Calculator
Agent failures don't add linearly — they compound. A 10-step workflow where each step succeeds 95% of the time doesn't give you 95% end-to-end success. It gives you 60%.
Adjust the sliders to see how quickly individual step accuracy translates into end-to-end collapse:
Work through the math: if a system handles 1,000,000 daily requests and 4% fail silently — plausible with a 25-step chain at 99.8% per step — that's 40,000 wrong answers daily, all at HTTP 200 OK. Your real rate will depend on chain length and step reliability. The point is: this never appears in a standard monitoring dashboard.
The 4-Layer Observability Control Plane
A production-grade agent observability control plane intercepts, processes, stores, and analyzes telemetry across four strictly separated layers. Each layer has different storage engines, different query patterns, and different latency requirements.
- 1
Layer 1 emits what and why
The application layer emits domain signals about decisions made. The framework emits structural signals about how orchestration ran. The runtime emits hardware signals about resource constraints.
- 2
Layer 2 decouples and enriches
The collection layer sits between the app and storage. It handles protocol translation, batching, signal enrichment, and — critically — tail-based sampling decisions.
- 3
Layer 3 stores by access pattern
Metrics go to time-series databases optimized for numerical aggregation. Logs go to full-text search engines. Traces go to distributed trace storage indexed by Trace ID.
- 4
Layer 4 correlates across signals
The key power: click a latency spike on a metrics dashboard, jump directly to the underlying trace, then filter logs by that exact Trace ID — without copy-pasting.
X-Ray Trace Inspector
This is what a real agent trace looks like. Click any span to expand its attributes. Notice how the HTTP 200 OK at the root level tells you nothing — you have to inspect the retrieval span to find the silent failure.
The retrieval step returned documents with similarity scores of 0.41 and 0.38 — both below the quality threshold of 0.70. The LLM generated a confident-sounding response using that garbage context. The HTTP status code reported 200 OK throughout.
Telemetry Standards: OTel vs OpenInference
Standard OpenTelemetry was built for HTTP endpoints and database queries. It has no native primitives for prompt tokens, vector similarity scores, or reasoning steps. OpenInference extends OTel specifically for AI systems.
Standard OTel Span
Captures: HTTP method, status code, URL, peer IP. Misses: token counts, model ID, prompt text, retrieval scores, retry logic.
OpenInference Span
Adds: llm.model_name, llm.token_count.*, tool.parameters, retrieval.documents.*.score, agent.retry_count.
Trace Context Propagation
W3C traceparent headers carry Trace ID + Span ID across service boundaries, sub-agents, and async calls — keeping the entire journey stitched together.
The 7 Span Primitives You Need
| Primitive | What It Stores | Why It Matters for Agents |
|---|---|---|
| Trace ID | 128-bit global request identifier | Ties every sub-agent call, tool, and DB query to one user request |
| Root Span | Full transaction from ingress to response | Your timer for end-to-end latency — the thing the user actually experiences |
| Child Spans | Named sub-tasks nested inside parent | RAG retrieval, LLM inference, tool call — each gets its own measurable unit |
| Span Attributes | Key-value metadata on each span | Where token counts, similarity scores, and model IDs live |
| Span Events | Timestamped point-in-time markers | Rate limit hits, retry attempts, stream chunk arrivals |
| Span Links | Causal references across separate trace contexts | Essential for async sub-agent delegation — unlike parent-child spans they cross trace ID boundaries |
| Context Propagation | Serialized Trace ID in HTTP/gRPC headers | Prevents trace fragmentation when requests cross service boundaries |
Span Links in Multi-Agent Systems
Parent-child spans share the same Trace ID. Span Links are different — they reference spans in a completely separate trace context. This is what makes async sub-agent delegation traceable: the Research Agent starts its own trace, but its root span links back to the Orchestrator span that spawned it.
Sampling Strategies: Head-Based vs Tail-Based
Storing 100% of production traces is cost-prohibitive. But choosing what to keep is critical — the wrong approach discards exactly the failures you need.
9 Directions Worth Exploring
Current observability platforms handle infrastructure metrics and distributed traces well. The nine areas below aren't necessarily missing from every product — some have partial solutions, others are emerging. Think of them as directions the field is still working through: good hackathon territory, or things to watch as products mature.
These are drawn from research and architectural thinking, not from testing specific production systems. They're worth watching as the agent observability space matures — and several make strong hackathon starting points.
Vendor Landscape
Four distinct platform archetypes exist. The right choice depends on how much control you need vs. how much operational overhead you can absorb.
| Archetype | Examples | Control | Agent-Native | Complexity |
|---|---|---|---|---|
| Self-Managed | OTel Collector + Prometheus + Jaeger | Maximum | Low — manual | High |
| Commercial Managed | Datadog, Dynatrace, New Relic | Moderate | Moderate — LLM wrappers | Low |
| Cloud-Native | AWS CloudWatch, GCP Operations | Moderate | Low — infra only | Low |
| Purpose-Built AI | Arize Phoenix, LangSmith, Braintrust | Moderate | Maximum — first-class | Low–Medium |
Start with a purpose-built AI platform (Arize Phoenix or LangSmith) for rapid time-to-value on agent-specific observability. Layer in OTel-based infrastructure telemetry alongside it rather than choosing one or the other. Never use only a general-purpose APM tool for agent observability — you will be completely blind to the failures that matter most.
Inspired by a session on Observability for AI Agents by Elena Karrer from MongoDB. The architecture synthesis, interactive tools, extended thinking, and 9 directions to explore are the author's own research and learning built on top of that foundation.