Skip to main content

Troubleshooting

This guide covers common issues encountered when running AgentVisor™, organized by symptom for quick diagnosis.

Sandbox Startup Failures​

gVisor not available​

Symptom: Error message runsc not found or gvisor sandbox unavailable.

Cause: The gVisor runtime (runsc) is not installed or not in PATH. gVisor is only available on Linux.

Fix:

  1. Verify you're on Linux — gVisor doesn't support macOS or Windows
  2. Install gVisor: https://gvisor.dev/docs/user_guide/install/
  3. Ensure runsc is in PATH or configure guest.runsc_path in your config: 
    guest:
      runsc_path: /usr/local/bin/runsc

Workaround: Use --sandbox=docker on macOS/Windows, or --sandbox=none for local development.

gVisor rootless preflight failed​

Symptom: Error at startup: gVisor rootless mode is enabled but cannot create user namespaces on this host.

Cause: The host is blocking creation of unprivileged user namespaces. gVisor rootless mode requires unshare(CLONE_NEWUSER) to succeed. Three things can block it:

  1. Kernel sysctl kernel.unprivileged_userns_clone=0 (Debian, older Ubuntu)
  2. Container seccomp profile blocking the syscall (Kubernetes and Docker default profiles)
  3. AppArmor restricting unprivileged user namespaces (Ubuntu 24.04+, GKE, AKS, Azure Linux 3)

Fix — Kubernetes:

securityContext:
  seccompProfile:
    type: Unconfined
  appArmorProfile:        # Kubernetes 1.30+
    type: Unconfined

On clusters older than 1.30, use the legacy annotation on the Pod:

metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/<container-name>: unconfined

Fix — Docker:

docker run \
  --security-opt seccomp=unconfined \
  --security-opt apparmor=unconfined \
  ...

Fix — Host sysctl:

sudo sysctl -w kernel.unprivileged_userns_clone=1
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

Cloud-specific notes:

  • GKE: Standard node pools on COS+containerd require both seccompProfile and appArmorProfile set to Unconfined.
  • AKS: Mariner/Azure Linux 3 nodes enforce AppArmor by default; set appArmorProfile: Unconfined rather than falling back to privileged: true.
  • EKS: Bottlerocket and Amazon Linux 2023 generally only need seccomp unconfined, but check cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns on the node.

Diagnose on the node:

# Check userns sysctl
cat /proc/sys/kernel/unprivileged_userns_clone   # should be 1 (or file absent)
cat /proc/sys/user/max_user_namespaces            # should be > 0

# Check AppArmor restriction (Ubuntu 24.04+)
cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns  # should be 0 (or file absent)

Workaround: Switch to privileged mode by setting AGENTVISOR_GUEST_ROOTLESS=false and securityContext.privileged: true. This trades the least-privilege posture for broader compatibility.

Docker socket permission denied​

Symptom: Error permission denied while trying to connect to the Docker daemon socket.

Cause: The current user lacks permission to access /var/run/docker.sock.

Fix:

  1. Add your user to the docker group: 
    sudo usermod -aG docker $USER
  2. Log out and back in (or run newgrp docker)
  3. Verify with docker ps

Alternative: Run with sudo (not recommended for development).

Guest health check timeout​

Symptom: Error guest health check failed: context deadline exceeded or startup timeout.

Cause: The guest runtime didn't become ready within the configured timeout. Common reasons:

  • Slow container image pull
  • Resource constraints (CPU/memory limits too low)
  • Missing dependencies in the guest image
  • Agent crashes during initialization

Fix:

  1. Increase the startup timeout: 
    guest:
      startup_timeout: 60s  # Default: 30s
  2. Check if the base image needs to be pulled: 
    docker pull ghcr.io/manetu/agentvisor/agentvisor-guest:latest-python
  3. Review agent logs for initialization errors: 
    agentvisor serve . --sandbox=none  # Test without sandbox first
  4. Increase resource limits if the agent is memory-intensive: 
    guest:
      memory_limit: 1073741824  # 1GB

Policy Evaluation Errors​

AUTHZ_TYPE not set​

Symptom: Error authorization configuration required: set authz.type (available: [allowall embedded http]).

Cause: The authorization provider type is not configured. This is a required setting.

Fix: Configure an authorization provider:

authz:
  type: embedded  # Or: http, allowall
  embedded:
    policy_domain_files: "/path/to/policy.yml"

Or via environment variable:

export AGENTVISOR_AUTHZ_TYPE=embedded
export AGENTVISOR_AUTHZ_EMBEDDED_POLICY_DOMAIN_FILES="/path/to/policy.yml"

For development without policies, use allowall (never in production):

export AGENTVISOR_AUTHZ_TYPE=allowall

Policy file not found​

Symptom: Error failed to load policy domain file: open /path/to/policy.yml: no such file or directory.

Cause: The path in policy_domain_files doesn't exist or isn't readable.

Fix:

  1. Verify the file exists: 
    ls -la /path/to/policy.yml
  2. Use absolute paths — relative paths resolve from the working directory
  3. For multiple files, separate with commas: 
    authz:
      embedded:
        policy_domain_files: "/etc/agentvisor/base.yml,/etc/agentvisor/custom.yml"

Tip: Every scaffolded project (agentvisor template create) ships a starter policy at policies/domain.yml:

export AGENTVISOR_AUTHZ_EMBEDDED_POLICY_DOMAIN_FILES="./policies/domain.yml"

Allow-all warning in production​

Symptom: The host runtime logs this warning at startup:

SECURITY: allowall authorization provider is active - ALL requests are permitted without policy evaluation. NEVER use in production.

Cause: The allowall authorization provider is configured, which disables all security controls.

Detect it: Grep logs for the stable fragment (not the full sentence, which may be reworded):

grep "allowall authorization provider is active" /path/to/agentvisor.log

A deployment is safe from this specific bypass only when this grep finds nothing.

Fix: Never use allowall in production. Configure a real authorization provider:

authz:
  type: embedded
  embedded:
    policy_domain_files: "/etc/agentvisor/policies/production.yml"

Authorization denied unexpectedly​

Symptom: Error authorization denied for action X on resource Y when the action should be allowed.

Cause: The policy doesn't grant the required permission, or the principal doesn't match what the policy expects.

Debug steps:

  1. Enable pretty-printed access logs for debugging:

    authz:
      embedded:
        access_log_output: stdout   # or logger, stderr, /path/to/file
        access_log_format: pretty   # indented JSON (use json for compact)

    Or via environment variable: AGENTVISOR_AUTHZ_EMBEDDED_ACCESS_LOG_OUTPUT=stdout AGENTVISOR_AUTHZ_EMBEDDED_ACCESS_LOG_FORMAT=pretty

    note

    In agentvisor exec mode the default is output: logger to keep the terminal clean. Use --verbose or --log-file to see access log entries in that mode.

  2. Check the logs to see the exact PORC (Principal, Operation, Resource, Context) being evaluated

  3. Verify the principal in the request matches what your policy expects

  4. Check MRN format — common resources:

    • HTTP: mrn:agentvisor:http:<host>/<path>
    • Tools: mrn:agentvisor:tool:<name>
    • MCP: mrn:agentvisor:mcp:<server_name>/<tool_name>
    • A2A: mrn:agentvisor:a2a:<agent_name> (agent) or mrn:agentvisor:a2a:<agent_name>/task/<task_id> (task)

Temporal Connectivity​

Connection refused​

Symptom: Error connection refused or failed to connect to Temporal at localhost:7233.

Cause: Temporal server isn't running or isn't accessible at the configured address.

Fix:

  1. Start Temporal for local development: 
    temporal server start-dev
  2. Or start via Docker Compose: 
    docker compose up -d temporal
  3. Verify connectivity: 
    temporal workflow list --namespace default
  4. If using Temporal Cloud, verify the target address includes port 7233: 
    temporal:
      target: my-namespace.account-id.tmprl.cloud:7233

Namespace not found​

Symptom: Error namespace not found: my-namespace.

Cause: The configured namespace doesn't exist in Temporal.

Fix:

  1. For local Temporal, create the namespace: 
    temporal operator namespace create my-namespace
  2. For Temporal Cloud, the namespace format is namespace.account-id: 
    temporal:
      namespace: my-namespace.abc123
  3. Verify available namespaces: 
    temporal operator namespace list

Temporal Cloud authentication failures​

Symptom: Error authentication failed or permission denied when connecting to Temporal Cloud.

Cause: API key or mTLS certificates are missing, expired, or misconfigured.

For API key authentication:

  1. Verify the API key is set (never put it in config files): 
    export AGENTVISOR_TEMPORAL_AUTH_API_KEY="your-api-key"
  2. Ensure the auth type is configured: 
    temporal:
      auth:
        type: api_key
  3. Check that the API key hasn't expired in Temporal Cloud UI

For mTLS authentication:

  1. Verify certificate files exist and are readable: 
    ls -la /path/to/client.crt /path/to/client.key
  2. Check certificate expiry: 
    openssl x509 -enddate -noout -in /path/to/client.crt
  3. Ensure the namespace matches the certificate: 
    temporal:
      target: my-namespace.account-id.tmprl.cloud:7233
      namespace: my-namespace.account-id
      auth:
        type: mtls
        mtls:
          cert_file: "/path/to/client.crt"
          key_file: "/path/to/client.key"

Proxy Credential Substitution​

Symbolic token not substituted​

Symptom: Upstream API returns 401 Unauthorized or invalid API key, and logs show the symbolic token (e.g., mav-tok-...) being sent instead of the real credential.

Cause: The credential resolver isn't matching the request. Common reasons:

  • destinations regex doesn't match the target host
  • The request uses http:// instead of https:// (credentials require https by default — set allow_insecure: true on the rule to opt in to plaintext)
  • The request targets a non-standard port (only the scheme's standard port matches: 443 for https, 80 for http)
  • The credential isn't configured for the header being used
  • Environment variable with real credential isn't set

Fix:

  1. Check that the destination regex matches the target host: 
    proxy:
      credentials:
        - name: anthropic-key
          destinations:
            - "api\\.anthropic\\.com"  # Note: dots must be escaped
  2. Verify the environment variable with the real API key is set: 
    echo $ANTHROPIC_API_KEY  # Should show the real key, not mav-tok-...
  3. Check logs for credential resolution messages

Destination regex not matching​

Symptom: Credential substitution works for some hosts but not others.

Cause: The regex pattern doesn't match the actual hostname. Patterns are auto-anchored (implicitly wrapped with ^ and $).

Common mistakes:

# WRONG: Missing escape for dots
destinations:
  - "api.anthropic.com"  # Matches "apiXanthropicYcom"

# CORRECT: Escaped dots
destinations:
  - "api\\.anthropic\\.com"  # Matches only "api.anthropic.com"

# WRONG: Partial match expected
destinations:
  - "anthropic"  # Won't match "api.anthropic.com"

# CORRECT: Use .* for partial matching
destinations:
  - ".*\\.anthropic\\.com"  # Matches "api.anthropic.com", "upload.anthropic.com"

Debug: Enable debug logging and watch for credential resolution messages:

AGENTVISOR_LOG_LEVEL=debug agentvisor serve .

Missing destinations field​

Symptom: Error destinations is required for proxy credential or credential applies to all hosts unexpectedly.

Cause: The destinations field is required to prevent credentials from leaking to unintended hosts.

Fix: Always specify which hosts should receive the credential:

proxy:
  credentials:
    - name: my-api-key
      guest_env_var: MY_API_KEY
      destinations:
        - "api\\.myservice\\.com"
        - "api\\.myservice\\.io"
      resolver:
        type: bearer_token
        source: env
        env_var: MY_API_KEY

Guest TLS Errors​

SSL certificate errors in agent​

Symptom: Agent code fails with SSL: CERTIFICATE_VERIFY_FAILED or unable to get local issuer certificate.

Cause: The agent isn't using the ephemeral CA certificate that AgentVisor generates for TLS interception.

Fix:

  1. Ensure SSL_CERT_FILE is set in the agent environment (AgentVisor sets this automatically)
  2. If using custom HTTP clients, configure them to use the system CA bundle: 
    import httpx
    # Uses SSL_CERT_FILE automatically
    client = httpx.Client()
  3. For requests library: 
    import os
    import requests
    # requests uses SSL_CERT_FILE when REQUESTS_CA_BUNDLE isn't set
    response = requests.get("https://api.example.com")

For unsandboxed mode (--sandbox=none): The CA certificate isn't injected automatically. Either:

  • Use --sandbox=docker for testing TLS behavior
  • Or disable TLS verification for local debugging only (never in production)

CA certificate not injected​

Symptom: SSL_CERT_FILE environment variable is empty or points to a non-existent file.

Cause: Running in --sandbox=none mode, which bypasses the guest runtime and CA injection.

Fix: This is expected behavior for --sandbox=none. The ephemeral CA is only needed when running through the TLS-intercepting proxy, which requires a sandbox.

Workaround for local testing:

  • Use --sandbox=docker to test with full TLS interception
  • Or configure your agent to skip TLS verification in development only

Certificate expiry warnings​

Symptom: Warning certificate expires soon or TLS errors after running for extended periods.

Cause: Per-host certificates have a default validity of 1 hour. For long-running sandboxes, certificates may expire.

Fix: These are environment-variable-only knobs, not YAML config keys. Increase certificate validity for long-running development sessions:

export AGENTVISOR_GUEST_CERT_VALIDITY=24h  # Default: 1h

The ephemeral CA has a default validity of 30 days:

export AGENTVISOR_GUEST_CA_VALIDITY=90d  # Default: 30d

MCP Gateway Issues​

Connection pool exhausted​

Symptom: Error connection pool exhausted or too many connections for MCP servers.

Cause: The number of concurrent principal-bound MCP connections exceeds the pool limit.

Fix: Increase the pool size:

mcp:
  pool:
    max_size: 200  # Default: 100

Alternative: Review if you need principal-bound credentials. Static credentials (bearer_token, api_key) share a single connection and don't consume pool slots.

MCP idle timeout​

Symptom: MCP tool calls fail intermittently with connection errors after periods of inactivity.

Cause: Idle connections are being closed but the agent expects them to remain open.

Fix: Adjust the idle timeout based on your usage pattern:

mcp:
  pool:
    idle_timeout: 15m  # Default: 5m

Note: Longer timeouts consume more resources. If agents use MCP servers sporadically, shorter timeouts with automatic reconnection may be more efficient.

MCP credential resolution failures​

Symptom: Error failed to resolve credentials for MCP server or token exchange failed.

For bearer_token resolver:

  1. Verify the environment variable is set: 
    echo $GITHUB_TOKEN
  2. Check the configuration matches: 
    mcp:
      servers:
        - name: github
          credentials:
            type: bearer_token
            source: env
            env_var: GITHUB_TOKEN  # Must match actual env var

For token_exchange resolver:

  1. Verify the exchange URL is accessible
  2. Check that the principal JWT is valid and not expired
  3. Ensure the audience matches what the token exchange server expects: 
    credentials:
      type: token_exchange
      exchange_url: "https://auth.example.com/token"
      audience: "github-mcp"  # Must match server's expected audience

A2A Gateway Issues​

A2A connection pool exhausted​

Symptom: Error connection pool exhausted or too many connections for A2A agents.

Cause: The number of concurrent principal-bound A2A connections exceeds the pool limit.

Fix: Increase the pool size:

a2a_gateway:
  pool:
    max_size: 200  # Default: 100

Alternative: Review if you need principal-bound credentials. Static credentials (bearer_token, api_key, no auth) share a single connection per agent and don't consume pool slots.

A2A idle timeout​

Symptom: A2A calls fail intermittently with connection errors after periods of inactivity.

Cause: Idle connections are being closed but the agent expects them to remain open.

Fix: Adjust the idle timeout based on your usage pattern:

a2a_gateway:
  pool:
    idle_timeout: 15m  # Default: 5m

Note: Longer timeouts consume more resources. If agents call a given A2A agent sporadically, shorter timeouts with automatic reconnection may be more efficient.

A2A credential resolution failures​

Symptom: Error failed to resolve credentials for A2A agent or token exchange failed.

For bearer_token resolver:

  1. Verify the environment variable is set: 
    echo $RESEARCH_AGENT_TOKEN
  2. Check the configuration matches: 
    a2a_gateway:
      agents:
        - name: research-agent
          credentials:
            type: bearer_token
            source: env
            env_var: RESEARCH_AGENT_TOKEN  # Must match actual env var

For token_exchange resolver:

  1. Verify the exchange URL is accessible
  2. Check that the principal JWT is valid and not expired
  3. Ensure the audience matches what the token exchange server expects

Upstream JSON-RPC error codes​

The A2A Gateway client maps a subset of A2A's implementation-defined JSON-RPC error codes (internal/host/a2a/types) to Go sentinel errors; the rest surface as a raw JSON-RPC error with the code and message the external agent supplied:

CodeMeaningGateway client behavior
-32001Task not foundWrapped as ErrTaskNotFound
-32002UnauthorizedWrapped as ErrUnauthorized
-32003ForbiddenWrapped as ErrForbidden
-32004ConflictNot wrapped — raw JSON-RPC error
-32005Service unavailableWrapped as ErrUnavailable
-32006Operation timed outWrapped as ErrTimeout
-32007Invalid task stateWrapped as ErrInvalidState
-32008Agent not foundNot wrapped — raw JSON-RPC error
-32009Message rejectedNot wrapped — raw JSON-RPC error

Standard JSON-RPC 2.0 codes (-32700 parse error, -32601 method not found, etc.) can also surface. agentvisor a2a probe's tasks/list stage specifically treats method-not-found as unsupported rather than a failure — an agent with no task-listing support is still a reachable, working agent.

Only unauthorized and forbidden map to the auth_error classification used by audit records, GET /ready, and probe output. Every other code above — including operation timed out and service unavailable — falls into the generic connection_error bucket, because classification pattern-matches on the error's text rather than the JSON-RPC code itself ("operation timed out" does not contain the substring "timeout"). If error_class reads connection_error, don't assume a network-layer failure — check the verbatim message with AGENTVISOR_LOG_LEVEL=info,a2a=debug,a2a.client=debug before drawing a conclusion.

Interpreting GET /ready​

GET /ready reports MCP and A2A connectivity under the mcp and a2a keys, each a ComponentStatus:

{
  "status": "degraded",
  "warnings": ["mcp_servers_unavailable"],
  "mcp": {
    "configured": 3,
    "connected": 2,
    "failed": 1,
    "principal_bound": 1,
    "failed_endpoints": [
      {"name": "github", "error": "auth_error", "phase": "startup", "since": "2026-08-10T18:00:00Z"}
    ]
  }
}
  • failed_endpoints[] — one entry per server/agent with a currently failed recorded outcome. error is the same coarse classification code (connection_refused, dns_error, tls_error, auth_error, timeout, credential_error, connection_error) used by agentvisor mcp probe / agentvisor a2a probe — never the raw error text, which stays host-side in the log. This array (and the failure detail inside it) is only returned to a caller MPE authorizes for agentvisor:system:ready-detail against mrn:agentvisor:system:ready; every other caller — including an unauthenticated one — sees the counts only, with failed_endpoints omitted.
  • phase — "startup" if the current failure streak began before the gateway manager finished starting, "runtime" if it began from a live call afterward. A runtime failure also raises the coarse mcp_servers_degraded_at_runtime / a2a_agents_degraded_at_runtime warning, visible even to a caller not authorized to see failed_endpoints.
  • since — an RFC3339 UTC timestamp marking the start of the current consecutive failure streak, not the first failure ever observed. A successful call clears it; if the endpoint then fails again, that's a new streak with a new since, not a resumption of the old one.
  • Observed, not probed — every count here comes from real traffic, never a background health check. A static endpoint is marked connected/failed from its real connection attempt at startup. A principal-bound endpoint (principal_passthrough, token_exchange) that no principal has ever called has no recorded state at all — it counts toward principal_bound but appears in neither connected nor failed until some principal actually exercises it. GET /ready never proactively dials an endpoint on your behalf; use agentvisor mcp probe / agentvisor a2a probe for that.
  • TTL decay — a runtime-phase failure with no confirming traffic (a success or another failure) for 5 minutes is dropped from failed_endpoints entirely — reported as neither failed nor connected, since there's no longer positive evidence of either. Startup-phase failures never decay this way; they persist until a later successful call clears them.
  • The invariant — configured is not guaranteed to equal connected + failed + principal_bound. An unused principal-bound endpoint is the gap: it's counted once, in principal_bound, but not yet in connected or failed.

Schema Extraction Issues​

schema_cli fails​

Symptom: Error during agent startup: failed to extract schema or python3 -m agentvisor.schema_cli returned error.

Cause: The schema extraction script can't import or analyze your agent code. Common reasons:

  • Missing dependencies
  • Import errors in agent code
  • Python version mismatch
  • Timeout (default: 30s)

Fix:

  1. Test schema extraction manually (bare invocation prints help and exits 1 — pass --export): 
    cd your-agent
    python3 -m agentvisor.schema_cli --export .
  2. Fix any import errors in your agent code
  3. Ensure all dependencies are installed: 
    pip install -r requirements.txt
  4. Check for circular imports or heavy initialization code that runs at import time

Schema fallback behavior​

Symptom: Warning falling back to langgraph.json names only or incomplete agent schemas.

Cause: Schema extraction failed or timed out, so AgentVisor falls back to just reading agent names from langgraph.json without full schema details.

Impact: The agent will still work, but:

  • API documentation will be incomplete
  • Input validation may be less strict
  • A2A Agent Cards will have minimal skill descriptions

Fix:

  1. Run schema extraction manually to see the error: 
    python3 -m agentvisor.schema_cli --export . --output schema.json
  2. Fix any issues and cache the schema: 
    mkdir -p .agentvisor
    python3 -m agentvisor.schema_cli --export . --output .agentvisor/schema.json

Schema cache staleness​

Symptom: Agent changes aren't reflected in API documentation or tool schemas.

Cause: AgentVisor uses cached schema from .agentvisor/schema.json when it's newer than langgraph.json.

Fix:

  1. Delete the cached schema: 
    rm .agentvisor/schema.json
  2. Or regenerate it: 
    python3 -m agentvisor.schema_cli --export . --output .agentvisor/schema.json
  3. Restart the AgentVisor server

Tip: During active development, delete the cache directory to always use dynamic extraction:

rm -rf .agentvisor/

Getting Help​

If your issue isn't covered here:

  1. Check the logs: Increase log verbosity with AGENTVISOR_LOG_LEVEL=debug
  2. Contact support: support@manetu.com — include logs, configuration (redact secrets), and steps to reproduce

See Also​