Skip to main content

Tracing Your Agent

When a run is slow or does something unexpected, logs tell you what was said and a trace tells you where the time went and what called what. In this module you send AgentVisor™'s traces to a local Jaeger, find the spans for one run, and add a span of your own around the model call so it appears in the same trace.

What You'll Learn​

  • Running Jaeger as a Compose service alongside the Basic tutorial's services
  • Enabling telemetry.tracing and agent span forwarding in agentvisor.yaml
  • Finding the trace for a run, and reading its span tree
  • Adding a custom span from agent code with agentvisor.trace()
  • Filtering traces by thread, and why each run is its own trace

Prerequisites​

This module continues in the project you finished in the Basic tutorial. If you skipped it, scaffold the same agent:

agentvisor template create langgraph/tutorial-agent
cd tutorial-agent

You need Docker running and jq. This module doesn't depend on Human-in-the-Loop and Streaming; do either first or neither.

Not the same as syscall tracing

This module is about OpenTelemetry traces of what the agent does. The Tracing concept page is about a different feature, gVisor's system-call tracing of the sandbox. For the OpenTelemetry side in depth, see Observability.

How It Works​

Agent (sandbox)                    Host runtime                 Jaeger
agentvisor.trace(...) ──spans──▶  re-exports with its own  ──▶  OTLP :4317
                                  spans (API, run, policy,
                                  checkpoints, proxy)            UI :16686

The host creates a span for each API request and each run, and hands the run's trace context to the sandbox. Spans the agent creates are forwarded to the host, which exports them through the same connection, so one run is one trace with both sides in it.

Step 1: Add Jaeger​

Jaeger's all-in-one image receives OTLP traces and serves a UI. Create jaeger.yml next to compose.yml:

# Jaeger all-in-one: receives OTLP traces and serves a UI.
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "4317:4317"    # OTLP gRPC (AgentVisor exports here)
      - "16686:16686"  # Jaeger UI
    environment:
      - COLLECTOR_OTLP_ENABLED=true

A second Compose file merges with the first, so Temporal and Ollama still start from compose.yml:

docker compose -f compose.yml -f jaeger.yml up -d

Open http://localhost:16686. The Jaeger UI loads, with nothing to show yet.

Step 2: Turn On Tracing​

Add a telemetry section to agentvisor.yaml, below the existing proxy and mcp sections:

telemetry:
  tracing:
    enabled: true
    protocol: grpc
    endpoint: localhost:4317
    tls:
      mode: disable          # the local collector has no TLS
    sample_rate: 1.0         # keep every trace while you learn
  agent_tracing:
    service_name_template: "agent-{agent_name}"
  • endpoint is where the host exports OTLP over gRPC. Jaeger listens on 4317.
  • tls.mode: disable is right for a collector on your own machine. Use a verifying mode for anything remote.
  • sample_rate is the fraction of traces kept. 1.0 keeps all; production systems usually lower it.
  • service_name_template names the service that the agent's own spans appear under. Without it they show up as unknown_service:python3.

Step 3: See the Host's Spans​

Start the agent and send a chat message:

agentvisor serve .
THREAD=$(curl -sX POST http://localhost:8090/threads | jq -r '.thread_id')

curl -sX POST "http://localhost:8090/threads/$THREAD/runs?wait=60s" \
    -H "Content-Type: application/json" \
    -d '{"input": {"messages": [{"role": "user", "content": "fetch https://httpbin.org/get"}]}}' \
    | jq '.status'

In Jaeger, choose the service agentvisor-host and click Find Traces. The newest trace is your run. Open it. The span tree is rooted at the API request:

POST /threads/{id}/runs
├── UpdateWorkflow:createRun
│   ├── StartActivity:InvokeRun
│   └── ...
└── run
    ├── guest.load_checkpoint
    ├── guest.check_connect_policy
    ├── guest.http_request            ← the fetch_url tool's request
    │   └── proxy.execute
    ├── guest.http_request_stream     ← the agent's call to Ollama
    │   └── proxy.execute_stream
    ├── guest.emit_stream_chunk
    ├── guest.put_writes
    └── guest.save_checkpoint

Your trace will have more spans than this sketch; the shape is what to look for. The run span is the agent's run. Under it, each guest.* span is one call the sandboxed agent made to the host, such as loading and saving checkpoints, asking policy whether it may connect, or sending an HTTP request through the proxy. Click run and look at its tags: agentvisor.thread.id, agentvisor.run.id, agentvisor.principal and agentvisor.run.attempt identify the run.

Without any agent change you can already answer "how long did the model call take?" (the guest.http_request_stream span), "did policy run before the call?" (guest.check_connect_policy) and "how much of the run was checkpointing?"

Step 4: Add a Span from the Agent​

The host can see calls the agent makes, but not what the agent is doing between them. Add a span around the model call to see that part.

The SDK extra that carries spans out of the sandbox goes in requirements.txt:

agentvisor[tracing]

In agent.py, import agentvisor and wrap the model call in chat:

import agentvisor
    conversation = [SystemMessage(content=SYSTEM_PROMPT), *messages]
    try:
        with agentvisor.trace("chat.llm_invoke", attributes={"model": OLLAMA_MODEL}):
            response = get_llm().invoke(conversation)
        logger.debug("LLM response received")
        return {"messages": [response]}

agentvisor.trace() is a context manager that opens a span for the block. It does nothing when tracing is off, so the same agent runs fine without Jaeger.

Restart the agent. The guest image is rebuilt on startup, so the new requirement is installed. Send another message:

THREAD=$(curl -sX POST http://localhost:8090/threads | jq -r '.thread_id')

curl -sX POST "http://localhost:8090/threads/$THREAD/runs?wait=60s" \
    -H "Content-Type: application/json" \
    -d '{"input": {"messages": [{"role": "user", "content": "Say hello."}]}}' \
    | jq '.status'

In Jaeger there is now a second service, agent-chatbot (the template's graph is named chatbot). Find its traces and open one: chat.llm_invoke is a child of the host's run span, in the same trace as the guest.* spans, with your model attribute on it. The span's duration is the time spent inside your model call, as the agent sees it.

Step 5: Find a Thread's Runs​

Every run is its own trace, so a conversation with several runs is several traces. That includes a paused run and its resume, if you did Human-in-the-Loop and Streaming. To find all of them, search by thread. In Jaeger's search form, select service agentvisor-host and put this in Tags:

agentvisor.thread.id=<your thread id>

Or query the Jaeger API from the command line:

curl -s "http://localhost:16686/api/traces?service=agentvisor-host&tags=%7B%22agentvisor.thread.id%22%3A%22$THREAD%22%7D" \
    | jq -r '.data[] | "\(.traceID)  \(.spans | length) spans"'

The run ID tag, agentvisor.run.id, matches a single run, and it is the same ID the Runs API returned.

Optional: Automatic Instrumentation of LangChain Calls​

agentvisor.trace() traces the code you wrap. OpenLLMetry instruments LangChain, LangGraph nodes and tool calls without wrapping anything, adding spans for each. To try it, add to requirements.txt:

traceloop-sdk>=0.33.0
opentelemetry-instrumentation-langchain

and initialize it once in agent.py, after import agentvisor:

from traceloop.sdk import Traceloop

Traceloop.init(disable_batch=False)

Restart and run again. Expect many more spans under agent-chatbot, including prompt and completion details. That is useful for debugging and worth reviewing before you send traces anywhere you don't control, since prompts can contain sensitive content.

Next Steps​

You can now follow a run from the HTTP request through policy, checkpointing and the model call.