Skip to main content

Tracing Agent

OTEL distributed tracing with OpenLLMetry auto-instrumentation and manual spans, forwarded from the sandbox to Jaeger (or any OTLP backend).

Difficulty: Intermediate

Quick Reference​

agentvisor template create langgraph/tracing-agent
cd tracing-agent
docker compose up -d
agentvisor serve .
# View traces at http://localhost:16686

What You'll Learn​

  • OpenLLMetry auto-instrumentation for LangChain LLM calls
  • Manual spans with agentvisor.trace() for custom instrumentation
  • AgentVisor span forwarding: agent spans → host → OTLP backend
  • Host-side telemetry configuration via agentvisor.yaml
  • Full distributed trace: Temporal workflow → agent LLM call

Architecture​

User -> Thread API -> AgentVisor Host -> Temporal Workflow
                            |
                     RunAgent Activity
                            |
                       Guest Sandbox
                            |
                       Agent (Python)
                        Traceloop.init()
                        agentvisor.trace()
                            |
                     Span Forwarding (gRPC)
                            |
                      Host OTLP Exporter
                            |
                          Jaeger

Trace Hierarchy​

ThreadWorkflow                      (host — Temporal workflow)
  RunAgent                          (host — Temporal activity)
    chat                            (host — AgentVisor node span)
      chat.llm_invoke               (agent — manual span)
        langchain.ChatOllama        (agent — OpenLLMetry auto-span)
          HTTP POST /api/chat       (agent — outbound HTTP span)

Key Code​

Tracing Initialization​

Unlike chatbot-agent where tracing is optional, this example makes it required:

import agentvisor  # calls setup_tracing() if TRACEPARENT is set

# Required, not optional: wrapped in try/except only to raise a clear
# RuntimeError (install instructions) instead of a bare ImportError.
try:
    from traceloop.sdk import Traceloop
except ImportError as e:
    raise RuntimeError(
        "tracing-agent requires 'agentvisor[tracing]'.\n"
        "Install with: pip install 'agentvisor[tracing]' opentelemetry-instrumentation-langchain"
    ) from e

Traceloop.init(disable_batch=False)

Manual Span​

Demonstrates custom instrumentation alongside OpenLLMetry auto-instrumentation:

with agentvisor.trace(
    "chat.llm_invoke",
    attributes={"message_count": len(conversation), "model": OLLAMA_MODEL},
) as span:
    response = llm.invoke(conversation)
    if span:
        span.set_attribute("response_length", len(response.content))

Host Configuration​

agentvisor.yaml enables tracing on the host and forwards agent spans:

telemetry:
  tracing:
    enabled: true
    protocol: grpc
    endpoint: localhost:4317
    tls:
      mode: disable
    sample_rate: 1.0
  agent_tracing:
    enabled: true
    service_name_template: "agent-{agent_name}"

Policy Highlights​

Agent spans flow through gRPC ForwardSpans, not direct HTTP to Jaeger. Only Ollama needs to be allowed:

# Only Ollama endpoint allowed
allow if {
    helpers.is_authenticated
    regex.match("^ollama(:\\d+)?(/.*)?$", http_target)
}

Viewing Traces​

After running the agent, open Jaeger at http://localhost:16686:

  1. Select service agentvisor-host or agent-chatbot
  2. Click Find Traces
  3. Click a trace to see the full span tree

Configuring Other Backends​

Swap Jaeger for any OTLP-compatible backend in agentvisor.yaml:

Langfuse:

telemetry:
  tracing:
    enabled: true
    protocol: http
    endpoint: https://cloud.langfuse.com/api/public/otel
    headers:
      Authorization: "Basic <base64(publicKey:secretKey)>"

Honeycomb:

telemetry:
  tracing:
    enabled: true
    protocol: http
    endpoint: https://api.honeycomb.io
    headers:
      x-honeycomb-team: "<api-key>"

Next Steps​