The question that exposes the observability gap in most multi-agent systems is simple: "What happened during invocation abc123?" If the answer requires cross-referencing logs from three different services, querying the provider's dashboard, and asking the on-call engineer to remember what was deployed at 14:37 last Tuesday — you don't have observability. You have logs.
Observability, in the sense the term is used in distributed systems, means you can understand the internal state of a system from its external outputs. For a multi-agent AI workflow, that means: from the telemetry alone, you can reconstruct the full execution trace — which agents fired, in what order, which routing decisions were made, which fallback tiers activated, how much token budget was consumed at each step, and whether any HITL gates opened.
Most AI teams have partial observability at best: they can see that an agent made a call and got a response, but they can't reconstruct the routing decision that sent it to a particular model, the fallback that activated silently in the background, or the budget enforcement that rejected a downstream invocation.
The trace structure problem
Standard distributed tracing, as implemented by OpenTelemetry, models execution as a tree of spans: a root span representing the overall operation, with child spans for each sub-operation. For a microservices architecture, this works well — each service creates a span, propagates the trace context via HTTP headers, and the trace tree reflects the service call graph.
Multi-agent AI workflows have a different structure. The trace isn't just a service call graph — it's a decision graph. The control plane made a routing decision: that decision should be a span with attributes documenting why that model was selected (policy applied, quality threshold, cost ceiling). The fallback chain activated: that should be a span documenting which tier, which trigger condition, and what the latency was on the failed primary attempt. The budget enforcement event fired: that should be a span documenting the estimated token count, the ceiling, and the action taken.
A trace structure for a multi-agent invocation that gives you actual observability looks like this:
Trace: workflow=document-analysis session=session-789
└─ Span: control-plane.route
attributes:
workflow: document-analysis
routing.policy: cost_then_quality
routing.model_selected: gpt-4o-mini
routing.reason: workload_type=classification
└─ Span: model.invoke (gpt-4o-mini)
attributes:
model: gpt-4o-mini
tokens.prompt: 1240
tokens.completion: 87
latency_ms: 423
status: success
└─ Span: control-plane.route
attributes:
routing.model_selected: gpt-4o
routing.reason: workload_type=synthesis quality_threshold=0.85
└─ Span: model.invoke (gpt-4o) [FAILED]
attributes:
model: gpt-4o
status_code: 429
latency_ms: 312
└─ Span: fallback.activate
attributes:
fallback.tier: secondary
fallback.model: claude-3-5-sonnet
fallback.trigger: status_code_429
└─ Span: model.invoke (claude-3-5-sonnet)
attributes:
model: claude-3-5-sonnet
tokens.prompt: 3820
tokens.completion: 445
latency_ms: 1890
status: successThis trace tells you everything. Without it, the fallback activation is invisible — you see a successful response from Claude, but you don't know that a GPT-4o call failed first, that it was a rate limit, or that the fallback chain was the reason the request succeeded at all.
The metrics you actually need
Prometheus metrics for multi-agent systems have a different shape than metrics for stateless APIs. The metrics that matter for AI orchestration are:
Routing distribution. What fraction of invocations are going to each model? orchvynt_routing_model_invocations_total labeled by workflow, model, and workload_type. This tells you if your routing policy is behaving as expected and flags shifts in workload distribution.
Fallback activation rate. What fraction of invocations are hitting fallback tiers? orchvynt_fallback_activations_total labeled by workflow, tier, and trigger_reason. A rising fallback rate is a leading indicator of provider health degradation — often detectable before the provider posts a status page incident.
Budget enforcement events. How many invocations are being intercepted by budget enforcement? orchvynt_budget_enforcement_total labeled by workflow and enforcement_type. A rising budget enforcement rate indicates either a workflow cost regression or a budget ceiling that's too tight for the actual workload.
HITL gate dwell time. How long do invocations wait at HITL gates? orchvynt_hitl_gate_dwell_seconds — a histogram labeled by gate_id. Long dwell times indicate under-staffed review queues. Combine with gate open rate to understand reviewer load.
p50/p95/p99 latency by tier. Not aggregate latency, but latency broken down by model and tier. orchvynt_model_latency_seconds labeled by model and provider. This is how you detect provider-level degradation before it causes fallback activations — p99 latency on a specific provider rising while error rate is still zero is a leading warning sign.
Structured event log: the compliance surface
Metrics and traces are operational observability: they help your on-call engineer understand what the system is doing right now. The structured event log is a different artifact: it's the compliance surface. It answers retrospective questions: what routing decision was made on this specific invocation on this specific date? Who reviewed the HITL gate for this invocation? When was the budget enforcement policy last changed?
The event log should be append-only, write-once, and exportable. Every routing decision, budget enforcement event, fallback activation, HITL gate open, HITL reviewer decision, and config change should be a row in this log with a stable schema.
For cloud-hosted deployments, this log should ship to a storage backend you control (S3, GCS, or a self-managed object store) — not just retained in the vendor's dashboard, which can lose historical data on tier downgrades or account changes. For self-hosted deployments, the log writes to local filesystem or your preferred storage backend with no external dependency.
Exporting to your existing stack
Most teams have an existing observability stack before they adopt an AI orchestration layer. The right approach is for the control plane to export to that stack — not to require teams to adopt a new observability tool.
For traces: OTLP export is the right protocol. Any OpenTelemetry-compatible backend (Jaeger, Tempo, Honeycomb, Datadog, Dynatrace) accepts OTLP. Configure the exporter endpoint and the traces flow in.
For metrics: Prometheus /metrics endpoint plus a Grafana dashboard that your team can adopt directly. Pre-built dashboards that cover the key metrics — routing distribution, fallback rate, budget enforcement events, HITL dwell time — reduce the setup burden from hours to minutes.
For structured events: JSON to S3/GCS, or a Kafka topic for streaming into your data warehouse. The schema is consistent and well-documented, so building downstream reporting queries is straightforward.
The on-call benefit
The practical test of observability is: when your on-call engineer gets paged at 3am, how long does it take them to understand what's happening?
With distributed orchestration logic and no centralized telemetry, the answer is often 30-90 minutes of log correlation, provider status page checking, and deployment history review. With a control plane that emits structured telemetry, the answer is typically 2-5 minutes: the Grafana dashboard shows the fallback activation rate spiking on a specific workflow, the trace for a failing invocation shows a 429 cascade, and the metric shows which provider is rate-limiting. The configuration change to increase the fallback threshold or switch primary providers is a single config update, no deployment required.
Good observability doesn't just reduce debugging time. It changes the shape of incidents: from "we don't know what's happening" to "we know exactly what's happening and we know which config knob to turn."