Agent Tracing

AgentVisor provides end-to-end distributed tracing for AI agents. When enabled, agent LLM calls, tool invocations, and custom spans are forwarded from the sandbox to the host and exported to your observability backend alongside AgentVisor's own host-side spans.

Overview

┌────────────────────────────────────────────────────────────────┐
│  Guest Sandbox                                                  │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Python Agent                                            │  │
│  │  OpenLLMetry / manual OTEL → AgentVisorSpanExporter      │  │
│  └──────────────────────────────┬───────────────────────────┘  │
└──────────────────────────────────┼─────────────────────────────┘
                                   │ ForwardSpans gRPC
┌──────────────────────────────────┼─────────────────────────────┐
│  Host                            ▼                             │
│  SpanForwarder → host OTLP exporter → Jaeger / Langfuse / etc  │
└────────────────────────────────────────────────────────────────┘

How it works:

The host injects TRACEPARENT (and optionally TRACESTATE) environment variables into the sandbox before starting the agent. This establishes the W3C trace context for the run.
The AgentVisor Python SDK reads these variables at startup and configures an OTEL TracerProvider that uses AgentVisorSpanExporter.
OpenLLMetry (or manual OTEL instrumentation) produces spans inside the sandbox. Those spans are automatically children of the host's activity span.
AgentVisorSpanExporter serializes spans as OTLP protobuf and forwards them through the ForwardSpans gRPC call to the host.
The host's SpanForwarder re-exports the spans via the same OTLP exporter used for host spans.

The result: a single unified trace in your backend showing both AgentVisor's orchestration spans and the agent's LLM/tool spans.

Prerequisites

Host tracing must be enabled (telemetry.tracing.enabled: true)
The Python SDK tracing extra must be installed inside the agent:
```
agentvisor[tracing]
```
This pulls in opentelemetry-sdk and opentelemetry-exporter-otlp-proto-common.

How TRACEPARENT Works

When the host starts an agent run, it injects two environment variables into the sandbox:

Variable	Description
`TRACEPARENT`	W3C Trace Context header value (e.g. `00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01`)
`TRACESTATE`	Optional W3C TraceState header value for vendor-specific extensions

These are set automatically — no agent code changes are needed.

The SDK's setup_tracing() function (called during SDK initialization) detects TRACEPARENT, parses it, and establishes the parent context. Agent spans created after this point are correctly linked as children of the host's activity span.

If TRACEPARENT is absent (e.g. host tracing is disabled), the SDK does nothing — tracing is silently skipped.

Using OpenLLMetry for Auto-Instrumentation

OpenLLMetry provides automatic instrumentation for popular LLM frameworks and providers. When installed alongside the AgentVisor SDK, LLM calls are traced transparently.

Installation

Add to your agent's requirements.txt:

agentvisor[tracing]
opentelemetry-instrumentation-langchain

Or for specific LLM providers:

opentelemetry-instrumentation-openai
opentelemetry-instrumentation-anthropic
opentelemetry-instrumentation-ollama

Setup

Add one call to your agent startup code before any LLM calls:

from traceloop.sdk import Traceloop

# Auto-instrument all LLM calls
# AgentVisor tracing is already configured by the SDK — Traceloop will
# use the existing TracerProvider set up by agentvisor.setup_tracing().
Traceloop.init(disable_batch=False)

note

agentvisor.setup_tracing() is called automatically during SDK import (via agentvisor.__init__). You do not need to call it manually — just import agentvisor or from agentvisor import ... before calling Traceloop.init().

What Gets Traced

With opentelemetry-instrumentation-langchain:

LangChain chain invocations and their inputs/outputs
LangGraph node executions
Tool calls with names and arguments
LLM completions with token counts

With provider-specific instrumentation (e.g. opentelemetry-instrumentation-ollama):

Individual LLM API calls
Prompt and completion content
Model name, token usage, latency

Manual Spans

For custom spans alongside auto-instrumentation:

import agentvisor

def process_document(doc):
    with agentvisor.trace("process_document", attributes={"doc_id": doc.id}):
        # ... processing ...
        return result

agentvisor.trace() is a no-op if tracing is not configured (e.g. in development without TRACEPARENT).

Configuring the Host

Enable host tracing in agentvisor.yaml:

telemetry:
  tracing:
    enabled: true
    protocol: grpc         # or "http" for HTTP-only backends
    endpoint: localhost:4317
    tls:
      mode: disable        # for local collectors without TLS
    sample_rate: 1.0

  agent_tracing:
    enabled: true          # default: true when host tracing is enabled
    # service_name_template: "agent-{agent_name}"

`service_name_template`

By default, agent spans preserve whatever service.name the agent set (via OTEL_SERVICE_NAME or the resource built by OpenLLMetry). You can override this to impose a consistent naming convention:

agent_tracing:
  service_name_template: "agent-{agent_name}"

The {agent_name} placeholder is replaced with the LangGraph graph name (e.g. chatbot, research_agent). This produces service.name = agent-chatbot in your backend, making it easy to filter agent traces.

Backend Setup Guides

Jaeger (gRPC OTLP)

Jaeger natively accepts OTLP/gRPC on port 4317. No extra configuration needed.

Start Jaeger (Docker):

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:latest

AgentVisor config:

telemetry:
  tracing:
    enabled: true
    protocol: grpc
    endpoint: localhost:4317
    tls:
      mode: disable

View traces: Open http://localhost:16686, select service agentvisor-host (or your configured service_name). Agent spans appear as children of the RunAgent activity span.

Grafana Tempo (gRPC OTLP)

Tempo's OTLP/gRPC receiver listens on port 4317 by default.

telemetry:
  tracing:
    enabled: true
    protocol: grpc
    endpoint: tempo:4317
    tls:
      mode: disable

Use Grafana's Explore view with the Tempo data source and TraceQL to query traces.

Langfuse (HTTP OTLP)

Langfuse is an LLM-native observability platform with first-class support for token cost analytics, prompt debugging, and evaluations. It accepts OTLP only over HTTP (not gRPC).

Why Langfuse for AI agents:

LLM call traces with token counts and cost estimates
Prompt/completion content for debugging
Session-level analytics across multiple runs
Evaluation tools for quality measurement

Get your API keys from the Langfuse dashboard → Settings → API Keys. You need the public key and secret key.

Compute the Basic Auth value:

echo -n "publicKey:secretKey" | base64

AgentVisor config (agentvisor.yaml):

telemetry:
  tracing:
    enabled: true
    protocol: http
    endpoint: https://cloud.langfuse.com/api/public/otel
    headers:
      Authorization: "Basic <base64-value>"
  agent_tracing:
    enabled: true
    service_name_template: "agent-{agent_name}"

Self-hosted Langfuse:

telemetry:
  tracing:
    enabled: true
    protocol: http
    endpoint: https://your-langfuse.example.com/api/public/otel
    headers:
      Authorization: "Basic <base64-value>"

tip

Set Authorization via environment variable rather than hardcoding in YAML:

export AGENTVISOR_TELEMETRY_TRACING_HEADERS="Authorization=Basic <base64-value>"

Honeycomb

Honeycomb accepts OTLP/HTTP with an API key header.

telemetry:
  tracing:
    enabled: true
    protocol: http
    endpoint: https://api.honeycomb.io
    headers:
      x-honeycomb-team: "YOUR_API_KEY"
      x-honeycomb-dataset: "agentvisor"

Datadog

Datadog Agent exposes an OTLP/gRPC receiver on port 4317 (enabled with DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT).

telemetry:
  tracing:
    enabled: true
    protocol: grpc
    endpoint: datadog-agent:4317
    tls:
      mode: disable

Alternatively, use the Datadog OTLP intake directly over HTTP:

telemetry:
  tracing:
    enabled: true
    protocol: http
    endpoint: https://trace.agent.datadoghq.com
    headers:
      DD-API-KEY: "YOUR_DD_API_KEY"

OpenTelemetry Collector

For production, routing through an OTel Collector gives you vendor flexibility, batching, filtering, and fan-out to multiple backends.

Collector config (otel-collector.yaml):

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlphttp/langfuse:
    endpoint: https://cloud.langfuse.com/api/public/otel
    headers:
      Authorization: "Basic <base64>"
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/langfuse, jaeger]

AgentVisor config (points to the collector):

telemetry:
  tracing:
    enabled: true
    protocol: grpc
    endpoint: otel-collector:4317
    tls:
      mode: disable

Example: Chatbot Agent with Tracing

This example extends the langgraph/chatbot-agent template (agentvisor template create langgraph/chatbot-agent) with OpenLLMetry instrumentation for Ollama.

requirements.txt:

langgraph>=0.2.0
langchain-core>=0.3.0
langchain-ollama>=0.2.0
pydantic>=2.0.0
agentvisor[tracing]
opentelemetry-instrumentation-langchain

agent.py additions (at the top, after imports):

import agentvisor  # ensures setup_tracing() is called

# Auto-instrument LangChain/LangGraph calls.
# Spans are forwarded to the host through AgentVisorSpanExporter.
from traceloop.sdk import Traceloop
Traceloop.init(disable_batch=False)

No other changes needed. With this in place:

Start AgentVisor with tracing enabled (pointing at Jaeger or Langfuse)
Run the chatbot agent and send a message
Each LLM call appears as a span under the RunAgent activity span in your backend

Trace Hierarchy

A typical trace looks like:

ThreadWorkflow                         [AgentVisor Temporal workflow]
  └── RunAgent (activity)              [AgentVisor host span]
        ├── chat (LangGraph node)      [OpenLLMetry LangChain span]
        │     └── ollama.completion    [OpenLLMetry Ollama span]
        └── ...

The parent-child relationship is established automatically via TRACEPARENT. All spans share the same trace ID, enabling correlation in your backend.

Troubleshooting

Spans not appearing in backend:

Verify telemetry.tracing.enabled: true in agentvisor.yaml
Check host logs for "tracing enabled" and "agent span forwarding enabled" messages
Confirm agentvisor[tracing] is installed in your agent's virtual environment
For Langfuse: verify the Basic Auth header — echo -n "pub:sec" | base64 and compare against your Langfuse API keys
For HTTP backends: check AGENTVISOR_TELEMETRY_TRACING_TLS_MODE — HTTPS endpoints need TLS (default verify-full), local HTTP endpoints need mode: disable

All spans show under one service name:

Use service_name_template: "agent-{agent_name}" to get per-agent service names
Or set OTEL_SERVICE_NAME in framework.env in mav-agent-config.yaml for a fixed name

TRACEPARENT is set but no spans forwarded:

Ensure opentelemetry-sdk is installed (pip show opentelemetry-sdk)
The SDK prints "AgentVisor tracing configured" at DEBUG level when setup succeeds — set AGENTVISOR_LOG_LEVEL=debug,agent=debug to see it

gRPC vs HTTP confusion:

Default is protocol: grpc (port 4317)
Langfuse, Honeycomb, and some others require protocol: http
When protocol: http, endpoint can be a full URL including path (e.g. https://cloud.langfuse.com/api/public/otel)

Overview​

Prerequisites​

How TRACEPARENT Works​

Using OpenLLMetry for Auto-Instrumentation​

Installation​

Setup​

What Gets Traced​

Manual Spans​

Configuring the Host​

service_name_template​

Backend Setup Guides​

Jaeger (gRPC OTLP)​

Grafana Tempo (gRPC OTLP)​

Langfuse (HTTP OTLP)​

Honeycomb​

Datadog​

OpenTelemetry Collector​

Example: Chatbot Agent with Tracing​

Trace Hierarchy​

Troubleshooting​

See Also​