Observability

AgentVisor™ integrates with OpenTelemetry for distributed tracing and Prometheus for metrics collection, providing visibility into agent behavior, request flows, and system performance.

Distributed Tracing (OTLP)

AgentVisor exports traces via the OpenTelemetry Protocol (OTLP). Traces capture the full lifecycle of agent requests, including HTTP proxy calls, policy evaluations, and the request flow through AgentVisor's components.

Configuration

Variable	Default	Description
AGENTVISOR_TELEMETRY_TRACING_ENABLED	false	Enable OTLP trace export
AGENTVISOR_TELEMETRY_TRACING_ENDPOINT	localhost:4317	OTLP collector endpoint
AGENTVISOR_TELEMETRY_TRACING_PROTOCOL	grpc	OTLP transport: grpc or http (use http for Langfuse and other HTTP-only backends)
AGENTVISOR_TELEMETRY_TRACING_HEADERS	-	Not settable via env var (it's a map[string]string field) — set telemetry.tracing.headers as a YAML map in the config file instead. Only applies when protocol: http.
AGENTVISOR_TELEMETRY_TRACING_SAMPLE_RATE	1.0	Trace sampling rate (0.0 = none, 1.0 = all)
AGENTVISOR_TELEMETRY_TRACING_TLS_MODE	verify-full	TLS trust mode for the OTLP connection: disable, require, verify-ca, verify-full. For local dev with collectors on localhost, set to disable.
AGENTVISOR_TELEMETRY_TRACING_TLS_CA_FILE / _CA_DATA	-	Trust a private CA for the OTLP collector instead of the system root pool
AGENTVISOR_TELEMETRY_AGENT_TRACING_ENABLED	true	Forward agent OTLP spans to the host exporter (requires host tracing enabled)
AGENTVISOR_TELEMETRY_AGENT_TRACING_SERVICE_NAME_TEMPLATE	-	Template for agent span service.name. Supports {agent_name} placeholder. When empty, the agent's own OTEL_SERVICE_NAME is preserved.
AGENTVISOR_TELEMETRY_SERVICE_NAME	agentvisor-host	Service name for telemetry (overridden by AgentVisor ID if set)
AGENTVISOR_TELEMETRY_ENVIRONMENT	development	Deployment environment label

Example

telemetry:
  service_name: my-agentvisor
  environment: production
  tracing:
    enabled: true
    endpoint: otel-collector:4317
    sample_rate: 0.1    # Sample 10% of traces
    tls:
      mode: verify-full

Span Attributes

Every application span carries contextual attributes you can use for filtering. The execution-context attributes attached to every span:

Attribute	Description
agent.id	Agent name for the current thread
agentvisor.thread.id	Thread ID
agentvisor.run.id	Run ID
execution.id	Temporal execution / activity ID linking spans across host-guest boundaries

agentvisor.principal is not on every span — it's only attached to the Temporal activity span for an agent invocation and to spans re-exported from the agent's own OTLP traces, and only when a non-anonymous principal is known for that execution.

agentvisor.run.attempt is likewise only on the per-run span created for an InvokeRun activity: the 1-indexed s-6ia7 restart-loop attempt number for that activity. Every attempt gets its own activity and hence its own span under the same run trace, so a restarted run's attempts are distinguishable by this attribute rather than by a new span per restart.

Additional per-operation attributes are emitted by individual handlers — for example, http.method and http.url on proxy spans, mcp.server and mcp.tool on MCP gateway spans, store.namespace and store.key on store spans, checkpoint.name on checkpoint spans, and tunnel.target on HTTPS tunnel spans.

Metrics (Prometheus)

AgentVisor exposes a Prometheus-compatible /metrics endpoint for scraping.

Configuration

Variable	Default	Description
AGENTVISOR_TELEMETRY_METRICS_ENABLED	false	Enable Prometheus metrics endpoint
AGENTVISOR_TELEMETRY_METRICS_ENDPOINT	/metrics	HTTP path for metrics scraping

Available Metrics

HTTP API:

Metric	Type	Description
agentvisor_http_requests_total	Counter	Total HTTP API requests
agentvisor_http_request_duration_seconds	Histogram	HTTP API request latency
agentvisor_http_requests_in_flight	Gauge	Currently processing HTTP requests

Agent-Host Control Channel:

Metric	Type	Description
agentvisor_grpc_requests_total	Counter	Total requests from sandboxed agents to AgentVisor
agentvisor_grpc_request_duration_seconds	Histogram	Request latency

HTTP Proxy:

Metric	Type	Description
agentvisor_proxy_requests_total	Counter	Proxied HTTP requests
agentvisor_proxy_request_duration_seconds	Histogram	Proxy request latency

Authorization:

Metric	Type	Description
agentvisor_authz_decisions_total	Counter	Authorization decisions (allowed/denied)
agentvisor_authz_latency_seconds	Histogram	Authorization evaluation latency

Agent Runs:

Metric	Type	Description
agentvisor_runs_total	Counter	Total agent run invocations
agentvisor_run_duration_seconds	Histogram	Agent run duration

Run Restarts (s-6ia7):

Metric	Type	Description
agentvisor_run_restarts_total	Counter	Restart attempts made by the bounded restart loop, by class
agentvisor_run_restart_exhausted_total	Counter	Restartable failures the loop gave up on (attempt budget, wall-clock budget, or deterministic-crash protection exhausted), by class

class is the faults.FailureClass string: CRASH/INFRA for a classified InvokeRun activity failure that triggered the restart or the give-up decision, or CAN for the unified restart model's CAN-triggered-interruption bookkeeping — a Continue-As-New that interrupted an active run is counted as a restart too, since it bumps the same attempt counter, but it can never appear on agentvisor_run_restart_exhausted_total: a CAN-triggered interruption always proceeds via Continue-As-New rather than giving up. A run that restarted at least once also surfaces attempt and last_failure on its REST/A2A/MCP status response.

PutWrites Fail-Open Drops:

Metric	Type	Description
agentvisor_put_writes_dropped_total	Counter	Pending-write entries dropped by the put_writes fail-open policy, by bound
agentvisor_put_writes_dropped_bytes_total	Counter	Bytes of pending-write values dropped by the put_writes fail-open policy, by bound

LangGraph's put_writes (mid-superstep pending writes) is fail-open by design: a request that exceeds a configured bound is silently truncated rather than rejected, since rejecting it would surface as a hard graph failure. These two counters are the only way to observe that data loss outside of a per-request log line at each drop site. The bound attribute identifies which limit was hit and where:

bound	Recorded by
count	The guest-facing validator's per-request write-count cap, HostService's own defense-in-depth write-count cap, and the workflow store's per-checkpoint write-count cap (three independent enforcement points share this bound name)
value_size	A single pending write's value exceeding the per-write size limit (guest validator and workflow store)
total_size	The cumulative size of a batch exceeding the total pending-writes size limit (guest validator and workflow store)
non_head	A put_writes call naming a checkpoint other than the current head (workflow store only — pending writes attached to a non-head checkpoint could never be read back)
can_drop	Continue-As-New unconditionally dropping all pending writes from the payload so CAN can always complete, regardless of accumulated pending-write size

Workflow-context drops (non_head, value_size/total_size from the workflow store, and can_drop) are recorded through Temporal's replay-safe metrics handler (workflow.GetMetricsHandler), not the OTel API directly — a raw meter increment would double-count on every workflow replay. AgentVisor bridges that handler to the same OTel meter automatically whenever AGENTVISOR_TELEMETRY_METRICS_ENABLED=true; no separate Temporal-side configuration is needed.

CAN Payload Budget:

Metric	Type	Description
agentvisor_can_pending_writes_retained_total	Counter	Pending-write entries carried across a Continue-As-New instead of dropped (the can_drop counterpart above, for the retained case)
agentvisor_can_pending_writes_retained_bytes_total	Counter	Bytes of pending-write values carried across a Continue-As-New instead of dropped
agentvisor_can_messages_values_shed_total	Counter	Continue-As-New cleared the run-output read model (not durable execution state) to recover headroom under the CAN payload budget. Never fires when there was nothing to clear — clearing nothing is not a shed event
agentvisor_can_payload_budget_exceeded_total	Counter	Continue-As-New failed the workflow execution outright because the estimated payload still exceeded the budget even after dropping pending writes and shedding the read model — the checkpoint's own state is itself too large. Alert on this: it means a thread's execution has stopped, not merely degraded

See Checkpoint Sizing and Limits: Continue-As-New Budget for the budget these four metrics are measured against — derived from temporal.max_checkpoint_bytes, conservative under Temporal's real 2 MiB per-payload ceiling by default, or growing to match an explicitly-raised ceiling once External Checkpoint Storage is enabled — and the order the two shedding levers run in. Like the PutWrites metrics above, all four are recorded through workflow.GetMetricsHandler for the same replay-safety reason.

Guest Size-Limit Rejections:

Metric	Type	Description
agentvisor_size_limit_rejections_total	Counter	Guest-side requests rejected for exceeding a size limit, by limit and operation

operation is the rejected RPC name (SaveCheckpoint, StorePut, StatePut, EmitStreamChunk, ForwardLog, or ForwardSpans); limit identifies which bound was hit: checkpoint_envelope, temporal_payload, store_value, state_value, stream_chunk, log_message, or resource_spans. The guest reports this via a metrics-only LogMessage (empty message, limit_exceeded set) sent directly to the host, bypassing the guest's own configured log level, so the counter fires even when the corresponding WARN would be suppressed. There is deliberately no companion _bytes counter here: unlike the PutWrites drops above, a size-limit rejection doesn't lose data — the request simply fails — so a summable bytes counter would misrepresent it as data loss. The rejection message itself stays generic regardless of which bound was hit — the guest process has no access to host config, so it never names a temporal.* config key or an AGENTVISOR_* env var; the specific remediation is host-log-only, in the same forwarded LogMessage's fields. See Checkpoint Sizing and Limits: How to tell you're near a limit.

Query Payload Limits:

Metric	Type	Description
agentvisor_query_payload_limit_exceeded_total	Counter	Temporal Query responses that failed because the response payload exceeded Temporal's per-payload size limit (TMPRL1103), by query

query is one of getState, get_checkpoint, or list_checkpoints — the checkpoint/state-read query sites where a TMPRL1103 failure is plausible. This fires when a thread's checkpoint/state has grown too large for Temporal to return from a Query at all (a distinct, read-path failure shape from every other metric on this page, which all cover a write-path rejection or shedding decision). The error returned to the REST/Agent Protocol caller only reports that the thread's state is too large to read back — it never names a temporal.* config key or a Prometheus metric name, since any caller of that API can see it, not just the operator; the specific fix for the deployment's own checkpoint-storage configuration is logged host-log-only. See Checkpoint Sizing and Limits: Payload Size Limits.

Checkpoint Storage (temporal.checkpoint_storage — External Checkpoint Storage):

Metric	Type	Description
agentvisor_checkpoint_storage_operations_total	Counter	Store/Retrieve calls to the checkpoint storage backend, by operation (store/retrieve) and result (success/error)
agentvisor_checkpoint_storage_operation_duration_seconds	Histogram	Store/Retrieve call duration, by operation
agentvisor_checkpoint_storage_payload_bytes	Histogram	Size of each payload stored to or retrieved from the backend, by operation
agentvisor_checkpoint_storage_digest_mismatch_total	Counter	Retrieved payloads whose recomputed SHA-256 digest did not match the digest recorded in its claim — a silent integrity failure if left unmonitored
agentvisor_checkpoint_storage_circuit_breaker_trips_total	Counter	Times the backend's circuit breaker tripped open
agentvisor_checkpoint_storage_circuit_breaker_state	Gauge	Current circuit breaker state: 0=closed, 1=open, 2=half_open
agentvisor_checkpoint_storage_cleanup_objects_scanned_total	Counter	Checkpoint storage objects examined by cleanup — recorded by both per-thread reclamation and the safety-net sweep
agentvisor_checkpoint_storage_cleanup_objects_reclaimed_total	Counter	Checkpoint storage objects deleted because their owning workflow execution left retention — recorded by both per-thread reclamation and the safety-net sweep
agentvisor_checkpoint_storage_cleanup_objects_skipped_total	Counter	Checkpoint storage objects examined but not reclaimed, by reason (live or liveness_check_error) — recorded only by the safety-net sweep; per-thread reclamation already confirms its target is dead before scanning, so it has nothing to skip

agentvisor_checkpoint_storage_operations_total{operation="retrieve"} re-counts on every replay-induced repeat of a Retrieve call, so it measures Store/Retrieve traffic volume rather than a count of distinct payloads — a pre-existing quirk of this instrument, not a bug in the metric above it. Every retrieved payload's digest is independently recomputed and checked before use regardless of whether the digest-mismatch counter is being watched. See External Checkpoint Storage: Operating it and Cleanup for how these metrics fit into day-to-day operation, and Checkpoint Sizing and Limits: How to tell you're near a limit for the payload-bytes histogram's role as an early warning signal.

MCP Gateway:

Metric	Type	Description
agentvisor_mcp_requests_total	Counter	Total MCP requests. Only covers requests through pooled, principal-bound connections (principal_passthrough/token_exchange credentials) — static servers (bearer_token, api_key, no auth) are not counted here today.

A2A Gateway:

Metric	Type	Description
agentvisor_a2a_requests_total	Counter	Total A2A requests, by agent/operation/status
agentvisor_a2a_request_latency_seconds	Histogram	A2A request latency
agentvisor_a2a_errors_total	Counter	Total A2A errors, by agent/operation/error type

Gateway Connection Pools (internal/host/pool, shared implementation for MCP and A2A — same metric family under each gateway's own prefix):

MCP metric	A2A metric	Type	Description
agentvisor_mcp_pool_connections	agentvisor_a2a_pool_connections	Gauge	Current number of open pooled connections
agentvisor_mcp_pool_connections_opened_total	agentvisor_a2a_pool_connections_opened_total	Counter	Total pooled connections opened
agentvisor_mcp_pool_evictions_total	agentvisor_a2a_pool_evictions_total	Counter	Total pooled connections evicted (LRU or idle timeout)
agentvisor_mcp_pool_cache_hits_total	agentvisor_a2a_pool_cache_hits_total	Counter	Total pool cache hits (existing connection reused)
agentvisor_mcp_pool_cache_misses_total	agentvisor_a2a_pool_cache_misses_total	Counter	Total pool cache misses (new connection created)
agentvisor_mcp_pool_credential_refreshes_total	agentvisor_a2a_pool_credential_refreshes_total	Counter	Total pooled connections evicted and recreated because the principal's JWT changed

Pool metrics only apply to principal-bound connections (the same subset agentvisor_mcp_requests_total covers) — static MCP servers and static A2A agents connect once at startup and never enter the pool.

Trace Propagation

AgentVisor propagates trace context across the host-guest boundary and to upstream services. Configurable input and output propagation formats allow integration with existing tracing infrastructure.

Supported Formats

Format	Description
tracecontext	W3C Trace Context (default)
baggage	W3C Baggage (default)
b3	Zipkin B3 single-header
b3multi	Zipkin B3 multi-header
jaeger	Jaeger native format
xray	AWS X-Ray format

Configuration

Variable	Default	Description
AGENTVISOR_TELEMETRY_PROPAGATION_INPUT	tracecontext,baggage	Formats to extract from incoming requests
AGENTVISOR_TELEMETRY_PROPAGATION_OUTPUT	tracecontext,baggage	Formats to inject into outgoing requests

AgentVisor ID Integration

When an AgentVisor ID is configured, it integrates with telemetry:

• Service name override: The AgentVisor ID is used as the OpenTelemetry service name, allowing you to distinguish multiple AgentVisor instances in traces
• Log attribution: All log messages include an agentvisor_id field
• RFC 8693 delegation claims: The ID appears in the outermost act.sub claim of PORC principal tokens
• Authorization enrichment: The ID is surfaced to authorization enrichment providers as agentvisor_id, letting Rego or HTTP enrichers incorporate instance identity into their decisions