Runtime Tracing

AgentVisor™ provides runtime tracing to monitor syscall activity within gVisor sandboxes. This gives you deep visibility into agent behavior for security monitoring, debugging, and compliance.

Not to be confused with Agent Tracing, OpenTelemetry distributed tracing of LLM calls and tool invocations — an unrelated mechanism that happens to share the word "tracing."

Overview

When agents run inside a gVisor sandbox, every system call they make is intercepted by gVisor's Sentry (userspace kernel). AgentVisor's trace system captures these syscall events, enriches them with context (thread ID, principal, run ID), classifies them by severity, and routes them to configurable output sinks.

Key Benefits

Security Monitoring

• Detect suspicious file access patterns (reading /etc/passwd, SSH keys, credentials)
• Alert on process execution (shells, interpreters, security tools)
• Monitor network connection attempts
• Track privilege escalation attempts

Debugging

• Understand what your agent is actually doing at the syscall level
• Correlate syscalls with specific threads, runs, and principals
• Identify performance bottlenecks from filesystem or network operations

Compliance & Audit

• Complete audit trail of agent system interactions
• Attribution to authenticated principals
• Structured logs for SIEM integration

Trace Categories

AgentVisor groups syscalls into categories for easier configuration:

Category	Syscalls	Description
file_access	open, openat, read, write, close, stat, fstat, lstat, access	File operations
process	execve, execveat, fork, clone, exit, exit_group	Process lifecycle
network	socket, connect, bind, listen, accept, sendto, recvfrom	Network operations
identity	setuid, setgid, setreuid, setregid, capset	Identity/privilege changes
filesystem	chdir, chroot, mount, umount, pivot_root	Filesystem operations

Severity Classification

Every trace event is assigned a severity level based on the syscall type and its arguments:

Severity	Description	Examples
Info	Normal operations	read, write, close, stat, mmap
Warning	Potentially concerning	socket, connect, unlink, chmod, kill
Alert	Requires immediate attention	execve, setuid, ptrace, mount, chroot

Static Rules

The base severity comes from the syscall itself:

openat → info (normal file operation)
connect → warning (network activity)
execve → alert (process execution)

Pattern Rules

Pattern rules can upgrade severity based on arguments (never downgrade):

openat("/etc/passwd") → alert (sensitive file)
openat("/tmp/cache.txt") → info (no pattern match)
connect("192.168.1.100:443") → warning (private network)
connect("93.184.216.34:443") → warning (external network)

Built-in Pattern Rules

AgentVisor includes 20 built-in pattern rules for common security concerns. These run unconditionally and only ever affect severity — they do not populate rule_matches or tags on the event. Those two fields are populated only by the optional Rego rules engine described below.

Sensitive Files (→ Alert)

• System auth: /etc/passwd, /etc/shadow, /etc/sudoers
• SSH keys: ~/.ssh/*, ssh_host_*_key
• Cloud credentials: ~/.aws/credentials, ~/.kube/config, ~/.docker/config.json
• Private keys: *.pem, *.key, *.p12, *.pfx, *.jks
• Browser data: ~/.mozilla/*/cookies, ~/.chrome/*/logins
• Crypto wallets: ~/.bitcoin/wallet.dat, ~/.gnupg/secring

Dangerous Executables (execve → Alert)

• Shells: /bin/bash, /bin/sh, /usr/bin/zsh
• Interpreters: /usr/bin/python*, /usr/bin/perl, /usr/bin/node
• Network tools: /usr/bin/curl, /usr/bin/wget, /usr/bin/ssh, /usr/bin/nmap
• Security tools: /usr/bin/sqlmap, /usr/bin/hydra, /usr/bin/john
• Package managers: /usr/bin/apt, /usr/bin/pip, /usr/bin/npm
• Privilege escalation: /usr/bin/sudo, /usr/bin/su, /bin/mount

System Directories (→ Warning)

• /etc/*, /proc/*, /sys/*

External Network (connect → Warning)

• Any non-private IP (not 10.x.x.x, 172.16-31.x.x, 192.168.x.x, 127.x.x.x)

Rego Rules Engine (Optional)

Enabling trace.rules.enabled: true layers a Rego policy evaluation on top of the static classifier above. It can upgrade severity, attach rule_matches/ tags to the event, and set an alert flag that floors severity at alert. A built-in default policy is available (trace.rules.use_default_policy: true), or bring your own via policy_files/policy_inline. See Rego Rules Engine in the Security Monitoring Guide for the full default rule ID list and config.

Event Data Model

Each trace event carries the following fields:

Field	Description
timestamp	When the syscall occurred
syscall	Syscall name (e.g., openat)
syscall_args	Syscall arguments
thread_id	Conversation thread UUID¹
principal	Authenticated user (JWT subject)¹
run_id	Current run ID¹
severity	info, warning, or alert
category	file_access, process, etc.
rule_matches	Rego rule IDs matched (empty unless trace.rules.enabled)
tags	Rego-assigned labels (empty unless trace.rules.enabled)
file_path	Resolved file path
remote_addr	Network destination (IP:port)

¹ Empty unless exactly one run is active at syscall time — see the attribution caveat under Output Sinks below.

Output Sinks

Trace events are routed to one or more output sinks:

Log Sink

Outputs events as structured log messages:

{
  "time": "2026-02-18T11:18:35Z",
  "level": "ERROR",
  "msg": "syscall event",
  "syscall": "openat",
  "category": "file_access",
  "severity": "alert",
  "thread_id": "550e8400-e29b-41d4-a716-446655440000",
  "run_id": "run-12345",
  "agent_id": "research-agent",
  "principal": "user@example.com",
  "process": "python3",
  "pid": 1234,
  "path": "/etc/passwd"
}

rule_matches is only present when trace.rules.enabled: true (see Rego Rules Engine). The log sink never emits tags at all, even when the Rego rules engine populates them — none of the log, store, webhook, or gRPC sinks carry tags; only the SSE stream does.

Severity is mapped to log levels:

• Alert → ERROR
• Warning → WARN
• Info → INFO (or DEBUG, configurable)

The log sink rate-limits itself (1000 events/sec, burst 100) to prevent log flooding; this is currently fixed and not exposed as config — see Rate Limiting for alternatives at high volume.

Metrics Sink

Exports Prometheus metrics on the standard /metrics endpoint for dashboards and alerting — see Prometheus Metrics for the full metric list and the pre-built Grafana dashboard. The core counter:

Metric	Type	Labels	Description
agentvisor_trace_events_total	Counter	syscall, category, severity, agent_id*	Total trace events

* agent_id is included by default; there is currently no config knob to disable it for high-cardinality environments. It may still be empty on a given event — see the Attribution Caveat below.

agentvisor_trace_rule_matches_total (labeled rule_name, severity) is a separate metric that only populates when the Rego rules engine is enabled.

OTEL Sink

Exports events as OpenTelemetry spans, correlating syscall activity with distributed traces.

gRPC Sink

Streams events to an external gRPC collector in real time, for consumers that need a push-based feed rather than scraping metrics or tailing logs.

Webhook Sink

Batches events and POSTs them to an HTTP endpoint (e.g. a SIEM ingestion API), with retry and circuit-breaker protection against a failing endpoint.

Store Sink

Persists events to a file-based store on disk for replay and audit-trail queries, with configurable retention and pruning. There is currently no SQLite (or other database) backend for trace events.

Real-Time Streaming (SSE)

Independent of the sinks above, GET /traces/events/stream lets an external client subscribe directly to the event stream over Server-Sent Events, filterable by thread/run/agent/severity/category/syscall. See Real-Time Streaming (SSE) for query parameters and example usage.

Attribution Caveat

thread_id, run_id, agent_id, and principal are populated only when exactly one agent run is active on this AgentVisor instance at the moment the syscall occurs — gVisor reports a container ID, but AgentVisor runs one sandbox per instance, so mapping a syscall to a specific run is ambiguous with zero or more than one concurrent run. In that case these fields are left empty rather than guessing. The same conservatism applies at run boundaries: an event received before the active run started (e.g. a tail event of the previous run still in flight) is left unattributed rather than credited to the wrong run. This applies uniformly across every sink, not just the log sink.

Ring Buffer

Events pass through a ring buffer before reaching sinks. This:

• Decouples event production from sink processing
• Prevents blocking if a sink is slow
• Handles bursts without losing critical events

Buffer behavior when full:

Policy	Description
oldest	Drop oldest events (default)
newest	Drop incoming events
block	Block until space available (may impact agent)

Security Considerations

What Data is Captured

Trace events capture syscall metadata, not payload data:

• File paths, not file contents
• Network addresses, not packet payloads
• Process arguments, not stdin/stdout

Privacy Implications

Consider the sensitivity of traced data:

• File paths may reveal user activity patterns
• Network destinations may indicate services accessed
• Process names and arguments may contain parameters

Use appropriate access controls on log storage and configure log retention policies.

Socket Security

The trace receiver uses a Unix domain socket within the gVisor sandbox, inaccessible from outside the sandbox. Events flow from gVisor's Sentry to the host runtime over the existing secure channel.

Relationship to Other Observability

Feature	Purpose	Data Source
Runtime Tracing	Syscall-level visibility	gVisor seccheck
Application Logs	Agent business logic	Python logging
Distributed Tracing	Request flow across services	OTLP spans
Metrics	Performance and health	Prometheus

Runtime tracing complements these by providing the lowest-level view of what agents actually do at the OS level, independent of what they log or report.

Platform Requirements

note

Runtime tracing is only available with the gvisor sandbox mode. Other sandbox modes (docker, none) do not have access to gVisor's seccheck mechanism. Setting trace.enabled: true under docker or none does not fail startup — it logs a warning and tracing is simply disabled.

Next Steps

• Configuration Reference - All trace configuration options
• Security Monitoring Guide - Operational guide for security teams