Skip to main content

Security Monitoring Guide

This guide covers how to use AgentVisor's runtime tracing for security monitoring, including configuration, alert setup, SIEM integration, and troubleshooting.

Prerequisites​

  • AgentVisor deployed with gVisor sandbox mode (tracing is gVisor-only)
  • Basic understanding of Runtime Tracing concepts
  • Access to log aggregation (optional: Prometheus, Grafana, SIEM)

Quick Start​

Enable tracing with minimal configuration:

# agentvisor.yaml
trace:
  enabled: true

Or via environment variable:

export AGENTVISOR_TRACE_ENABLED=true

This enables tracing with default settings:

  • Categories: file_access, process
  • Log sink: enabled at info level
  • Buffer: 10,000 events, oldest drop policy

For security monitoring, enable all relevant categories and include context fields:

trace:
  enabled: true
  categories:
    - file_access    # File operations
    - process        # Process execution
    - network        # Network connections
    - identity       # Privilege changes
    - filesystem     # Mount/chroot operations
  context_fields:
    - container_id   # Correlate with gVisor container
    - process_name   # Which process made the syscall
    - credentials    # UID/GID context
    - cwd            # Working directory
  buffer:
    size: 50000      # Larger buffer for high-activity agents
    drop_policy: oldest
  sinks:
    log:
      enabled: true
      level: info    # Log all severities (info, warning, alert)
    metrics:
      enabled: true  # Enable Prometheus metrics

Understanding Severity Levels​

Events are classified into three severity levels:

Info​

Normal operations that don't require attention:

{"level":"INFO","msg":"syscall event","syscall":"read","severity":"info","path":"/app/data.json"}
  • File reads/writes to application directories
  • Memory operations (mmap, brk)
  • Timer and event operations

Warning​

Potentially concerning activity worth monitoring:

{"level":"WARN","msg":"syscall event","syscall":"connect","severity":"warning","remote_addr":"93.184.216.34:443"}
  • Network connections (especially external)
  • File deletions or permission changes
  • Process creation (non-exec)
  • Signal operations

Alert​

Security-critical events requiring immediate attention:

{"level":"ERROR","msg":"syscall event","syscall":"openat","severity":"alert","path":"/etc/passwd"}
  • Process execution (execve)
  • Sensitive file access
  • Privilege escalation attempts
  • Container escape attempts

Severity classification above is always active, via a built-in static classifier (syscall type + argument pattern matching). It does not populate rule_matches or tags — those two fields only appear when the Rego rules engine is enabled.

Configuring Alert Thresholds​

Log Sink Filtering​

trace.sinks.log.level does not filter which trace severities reach the log sink — every event that passes trace.categories is still evaluated. What it actually controls is the log level used for info-severity trace events specifically: warning-severity events always log at WARN and alert-severity events always log at ERROR, regardless of this setting (internal/host/trace/sinks/log.go).

trace:
  sinks:
    log:
      enabled: true
      level: warn    # info-severity events log at WARN instead of INFO; warning/alert are unaffected

Trace severity → log level mapping:

Trace SeverityLog LevelConfigurable via trace.sinks.log.level?
infoThis setting (default INFO)Yes
warningWARNNo — always WARN
alertERRORNo — always ERROR

To actually drop lower-severity events from output, raise the process-wide log level instead (AGENTVISOR_LOG_LEVEL=warn suppresses anything logged at INFO, including relabeled info-severity trace events, while warning and alert events still appear).

Rate Limiting​

The log sink includes built-in rate limiting to prevent log flooding:

  • Default: 1,000 events/second
  • Burst capacity: 100 events

Events exceeding the rate limit are silently dropped as they occur; a single summary of the total dropped count is logged once, when the sink closes (e.g. at shutdown) — not periodically while running.

Common Alert Patterns​

The patterns below drive severity classification unconditionally (built into the static classifier). The rule_matches and tags fields shown in the examples require trace.rules.enabled: true with the default policy (see Rego Rules Engine) — without it, these events still classify as alert/warning, just without the rule_matches/tags fields attached.

Detecting Sensitive File Access​

Built-in patterns alert on access to:

Authentication files:

/etc/passwd, /etc/shadow, /etc/sudoers, /etc/group

SSH keys and config:

~/.ssh/*, ssh_host_*_key

Cloud credentials:

~/.aws/credentials, ~/.kube/config, ~/.docker/config.json

Private keys:

*.pem, *.key, *.p12, *.pfx, *.jks

Example alert:

{
  "level": "ERROR",
  "msg": "syscall event",
  "syscall": "openat",
  "severity": "alert",
  "path": "/home/user/.ssh/id_rsa",
  "rule_matches": ["sensitive-credential-file"],
  "principal": "user@example.com",
  "agent_id": "research-agent"
}

Detecting Shell Execution​

Every execve/execveat syscall is unconditionally classified alert by the static classifier, regardless of which binary is run — so any shell execution is already flagged:

{
  "level": "ERROR",
  "msg": "syscall event", 
  "syscall": "execve",
  "severity": "alert",
  "path": "/bin/bash",
  "rule_matches": ["shell-execution"],
  "process": "python3"
}

The rule_matches: ["shell-execution"] tag is narrower: it requires trace.rules.enabled: true with the default policy, and only fires for an exact absolute-path match against /bin/{sh,bash,zsh,fish,csh,tcsh,ksh,dash} or the same names under /usr/bin/. A shell installed elsewhere (e.g. /opt/homebrew/bin/bash, a symlink, or busybox sh) still classifies as alert via the unconditional execve rule above, but won't carry the shell-execution rule match.

Detecting Network Tools​

Network reconnaissance and data exfiltration tools:

{
  "level": "ERROR",
  "msg": "syscall event",
  "syscall": "execve", 
  "severity": "alert",
  "path": "/usr/bin/curl",
  "rule_matches": ["network-tool-execution"]
}

Covered tools: curl, wget, nc, netcat, ssh, scp, rsync, nmap

Detecting External Connections​

Connections to non-private IPs are flagged:

{
  "level": "WARN",
  "msg": "syscall event",
  "syscall": "connect",
  "severity": "warning", 
  "remote_addr": "142.250.185.78:443",
  "rule_matches": ["external-network-connection"]
}

Private ranges (not flagged): 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 127.0.0.0/8

Rego Rules Engine (Optional)​

By default, severity classification is purely static (syscall type + argument pattern matching, described above) and events never carry rule_matches or tags. Enabling trace.rules.enabled: true layers a Rego policy evaluation on top: it can upgrade severity, set an alert flag (which floors severity at alert regardless of the policy's reported severity), and attach rule_matches/tags to the event.

trace:
  enabled: true
  rules:
    enabled: true
    use_default_policy: true   # built-in baseline policy (see below)
    # policy_files:             # additional/overriding policy files, merged
    #   - /etc/agentvisor/trace-policy.rego
    # policy_inline: ""         # inline policy string, merged in as well

Each event is evaluated with a ~50ms timeout. On policy error or timeout, the static classification stands — a broken or slow policy degrades to baseline behavior rather than blocking event processing.

Default Policy Rule IDs​

The built-in default policy (use_default_policy: true) populates rule_matches with these IDs:

Rule IDTrigger
sensitive-system-file/etc/passwd, /etc/shadow, /etc/sudoers, /etc/group
sensitive-credential-fileSSH keys, cloud credentials, private keys, browser data, crypto wallets
sensitive-directoryAccess under /etc/, /proc/, /sys/ (not already covered above)
shell-executionexecve/execveat of a shell binary
scripting-interpreterexecve/execveat of python/perl/ruby/node/php
network-tool-executionexecve/execveat of curl/wget/nc/nmap/etc.
security-tool-executionexecve/execveat of sqlmap/hydra/john/etc.
system-modification-toolexecve/execveat of package managers, sudo/su/mount/chroot
external-network-connectionconnect to a non-private, non-localhost address
privilege-escalationsetuid/setgid/setresuid/capset/etc.
container-escape-vectorchroot/pivot_root/mount/umount/umount2
process-injectionptrace/process_vm_readv/process_vm_writev

The same policy also attaches tags (file-access, network, process, identity, filesystem, sensitive, execution) for coarser filtering, and sets alert: true (floors severity at alert) for privilege escalation, container escape, and process injection regardless of the computed severity.

Custom policies can extend or override this via policy_files/policy_inline. Write additional rules in the same agentvisor.trace Rego package, populating the rule_matches, tags, and alert outputs the same way the built-in rules above do — those three fields make up the full output contract.

SIEM Integration​

Log Format​

Trace events are output as structured JSON, ready for SIEM ingestion:

{
  "time": "2026-02-18T14:30:00Z",
  "level": "ERROR",
  "msg": "syscall event",
  "syscall": "openat",
  "category": "file_access",
  "severity": "alert",
  "thread_id": "550e8400-e29b-41d4-a716-446655440000",
  "run_id": "run-abc123",
  "agent_id": "research-agent",
  "principal": "alice@example.com",
  "process": "python3",
  "pid": 1234,
  "path": "/etc/passwd",
  "rule_matches": ["sensitive-system-file"],
  "event_time": "2026-02-18T14:30:00.123456Z"
}

rule_matches (and tags) are only present when trace.rules.enabled: true (see Rego Rules Engine); omit them from SIEM parsing rules if you run with the static classifier only.

Key Fields for SIEM Rules​

FieldDescriptionSIEM Use
severityinfo, warning, alertTrigger rules on alert
syscallSyscall nameFilter by operation type
categoryEvent categoryGroup related events
principalAuthenticated userAttribute to user
agent_idAgent identifierScope to specific agent
thread_idConversation threadCorrelate within session
rule_matchesMatched Rego rule IDs (requires trace.rules.enabled: true)Identify alert reason
pathFile path (if applicable)Filter by resource
remote_addrNetwork destinationFilter by destination

Example SIEM Rules​

Splunk:

index=agentvisor severity=alert
| stats count by principal, agent_id, syscall, path
| where count > 5

Elastic:

{
  "query": {
    "bool": {
      "must": [
        {"term": {"severity": "alert"}},
        {"range": {"@timestamp": {"gte": "now-1h"}}}
      ]
    }
  },
  "aggs": {
    "by_principal": {"terms": {"field": "principal"}}
  }
}

Datadog:

service:agentvisor severity:alert | stats count by principal,agent_id

Real-Time Streaming (SSE)​

GET /traces/events/stream streams live trace events as Server-Sent Events — useful for a security dashboard or ad hoc curl monitoring without waiting on a log/metrics scrape interval. It requires trace.enabled: true; with tracing disabled it returns 503.

curl -N "https://agentvisor-host:8090/traces/events/stream?severity=warning,alert"

Query parameters (all optional, all combinable):

ParameterDescription
thread_idFilter to a specific thread
run_idFilter to a specific run
agent_idFilter to a specific agent
severityComma-separated: info, warning, alert
categoriesComma-separated: file_access, process, network, identity, filesystem
syscallsComma-separated syscall names
rate_limitPer-client override, events/second (capped at 10000)

The stream sends a connected event on open, periodic heartbeat events (reporting how many events this client had rate-limited since the last heartbeat), trace events carrying the JSON payload, and a done event if the server-side subscription closes. Concurrent subscribers are capped (503 once the limit is reached).

Attribution caveat: thread_id, run_id, agent_id, and principal are only populated when exactly one agent run is active on this AgentVisor instance at the moment the syscall occurs. gVisor reports a container ID, but AgentVisor runs exactly one sandbox per instance, so container-to-run mapping degrades to "the currently active run" — which is ambiguous with zero or more than one concurrent run. In either case these fields are left empty on the event rather than guessing, to avoid mis-attributing a syscall to the wrong principal. The same conservatism applies at run boundaries: an event received before the active run started (e.g. a tail event of the previous run still in flight) is left unattributed rather than credited to the wrong run. This applies to every sink, not just SSE: the log, metrics, webhook, gRPC, and store sinks all see the same unattributed events.

Prometheus Metrics​

When the metrics sink is enabled (trace.sinks.metrics.enabled: true), the following metrics are exported on the standard /metrics endpoint (see AGENTVISOR_TELEMETRY_METRICS_ENABLED):

MetricTypeLabelsDescription
agentvisor_trace_events_totalCountersyscall, category, severity, agent_id*Total trace events
agentvisor_trace_events_dropped_totalCounterreasonEvents dropped (buffer full, sink error, rate limited)
agentvisor_trace_rule_matches_totalCounterrule_name, severityRule matches — only emitted when trace.rules.enabled: true
agentvisor_trace_processing_duration_secondsHistogramsyscall, category, severity, agent_id*Time from event timestamp to sink emission
agentvisor_trace_rule_evaluation_duration_secondsHistogram-Rego rule evaluation latency
agentvisor_trace_buffer_sizeGauge-Current number of events in the buffer
agentvisor_trace_buffer_capacityGauge-Configured buffer capacity
agentvisor_trace_buffer_utilization_ratioGauge-buffer_size / buffer_capacity (0.0-1.0)

* agent_id is included by default; there is currently no config knob to disable it for high-cardinality environments. thread_id is never included as a metric label (unbounded cardinality) — use the log, store, or webhook sink for thread-level analysis.

There is no separate "alerts" metric — filter agentvisor_trace_events_total by severity="alert", or use agentvisor_trace_rule_matches_total when the Rego rules engine is enabled (see Rego Rules Engine).

Grafana Dashboard​

Import the pre-built trace-dashboard.json rather than hand-rolling panels — it already covers event rate by severity/category, buffer utilization, top agents/syscalls, rule matches, and processing latency percentiles using the metric names above. It requires Grafana 9+ and a Prometheus datasource scraping AgentVisor's /metrics endpoint.

To import it, open Grafana, go to Dashboards → Import, upload the file, and select your Prometheus datasource. To provision it instead, add a provider pointing at the directory containing the downloaded file:

# /etc/grafana/provisioning/dashboards/agentvisor.yaml
apiVersion: 1
providers:
  - name: 'AgentVisor'
    orgId: 1
    folder: 'AgentVisor'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /path/to/downloaded/dashboard

Ad hoc queries, if you need them outside the dashboard:

Alert rate:

sum(rate(agentvisor_trace_events_total{severity="alert"}[5m]))

Alerts by agent:

sum by (agent_id) (rate(agentvisor_trace_events_total{severity="alert"}[5m]))

Top alerting syscalls:

topk(10, sum by (syscall) (increase(agentvisor_trace_events_total{severity="alert"}[1h])))

Buffer utilization:

agentvisor_trace_buffer_utilization_ratio

Alert Rules​

Prometheus Alertmanager:

groups:
  - name: agentvisor-security
    rules:
      - alert: HighAlertRate
        expr: sum(rate(agentvisor_trace_events_total{severity="alert"}[5m])) > 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High rate of security alerts"

      # Requires trace.rules.enabled: true (rule_name labels come from the
      # Rego rules engine, not the static classifier).
      - alert: SensitiveFileAccess
        expr: increase(agentvisor_trace_rule_matches_total{rule_name=~"sensitive.*"}[1m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "Sensitive file accessed by agent"

      - alert: ShellExecution
        expr: increase(agentvisor_trace_rule_matches_total{rule_name="shell-execution"}[1m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "Shell execution detected in agent"
Fine-grained, rule-driven alerting is a trace-rules concern, not an MPE concern

MPE PolicyDomains authorize actions (HTTP endpoints, tool calls, and similar requests) and their input.resource never carries a file path or other syscall argument — a Rego rule written against input.resource.path cannot match a trace event and will never fire. Trace events are classified by a separate Rego engine instead (see Rego Rules Engine), whose input document does carry syscall arguments (input.args.path, input.args.remote_addr, etc.) and which populates the rule_matches/tags/alert fields used by the Alertmanager rules above.

Troubleshooting​

Events Not Appearing​

Check tracing is enabled:

# Should show trace config in startup logs
AGENTVISOR_LOG_LEVEL=debug agentvisor serve ./my-agent 2>&1 | grep -i trace

Verify gVisor sandbox:

# Tracing only works with gVisor
agentvisor serve ./my-agent --sandbox=gvisor

Check sink configuration:

trace:
  enabled: true
  sinks:
    log:
      enabled: true  # Must be true
      level: info    # Level for info-severity events; does not filter warning/alert

If events still don't appear, check the process-wide log level (AGENTVISOR_LOG_LEVEL) — trace.sinks.log.level only chooses which log level info-severity trace events are emitted at, not whether they are emitted. A global level above that (e.g. AGENTVISOR_LOG_LEVEL=warn) will still suppress them.

Missing Events​

Expand categories:

trace:
  categories:
    - file_access
    - process
    - network
    - identity
    - filesystem

Check buffer size:

trace:
  buffer:
    size: 50000  # Increase for high-activity agents

Check drop policy:

trace:
  buffer:
    drop_policy: oldest  # Keeps recent events

Too Many Events​

Increase log level:

trace:
  sinks:
    log:
      level: warn  # info-severity events log at WARN instead of INFO

This relabels info-severity events to WARN; it does not drop them. To actually reduce log volume, also raise AGENTVISOR_LOG_LEVEL above INFO so the relabeled events (and any other INFO-level logging) are filtered out by the logger itself.

Narrow categories:

trace:
  categories:
    - process   # Only process events

Use specific syscalls:

trace:
  syscalls:
    - execve
    - openat
    - connect

Debug Logging​

Enable debug logging to see pipeline internals:

AGENTVISOR_LOG_LEVEL=debug agentvisor serve ./my-agent

Look for:

level=debug msg="received event" type=SYSCALL timestamp=...
level=debug msg="enrichment failed" error="..."
level=debug msg="rule evaluation failed, keeping static classification" syscall=openat error="..."

enrichment failed and (when trace.rules.enabled: true) rule evaluation failed are the two messages to watch for — they indicate a syscall event made it through the pipeline but lost context attribution or rule classification respectively. The static classifier itself does not log per-event decisions (only sink construction is logged, at startup).

Performance Considerations​

Buffer Sizing​

Agent ActivityRecommended SizeDrop Policy
Low (simple chat)10,000oldest
Medium (tool use)25,000oldest
High (file processing)50,000+oldest

Sink Performance​

SinkLatencyUse Case
Log~1msDevelopment, low volume
Metrics~0.1msProduction dashboards
OTEL~5msDistributed tracing
gRPCNetwork-boundReal-time external consumers
WebhookNetwork-bound, batchedSIEM / security dashboard integration
StoreDisk-bound, bufferedAudit trail, post-hoc query/replay

Rate Limiting​

The log sink's rate limit (1,000 events/second, burst 100 — see Rate Limiting above) is currently fixed and not exposed as config; there is no trace.sinks.log.rate_limit key. For high-volume scenarios, prefer narrowing categories/syscalls, raising sinks.log.level to skip info events, or routing to the metrics/store/webhook sinks instead of the log sink.

Next Steps​