Scaling and Resilience

This guide covers how to scale AgentVisor™ deployments and ensure high availability for production workloads.

Architecture Foundation

AgentVisor achieves resilience through its integration with Temporal. Understanding this architecture is key to scaling effectively.

Temporal as the Resilience Backbone

All conversation state (threads, runs, checkpoints) is stored in Temporal workflows, not in AgentVisor instances. This means:

• Stateless nodes: AgentVisor instances hold no persistent state
• Any node can handle any request: Load balancers don't need sticky sessions
• Durable workflow state: If a node dies mid-request, the thread's workflow state survives in Temporal and a new node can pick up the next request — the in-flight run itself now restarts automatically, bounded, rather than always failing outright (see Node Failure Handling below)
• Workflow affinity: Temporal handles routing to the correct workflow context

┌─────────────────────────────────────────────────────────────────┐
│                        Load Balancer                            │
└─────────────────────────────────────────────────────────────────┘
                                 │
            ┌────────────────────┼────────────────────┐
            ▼                    ▼                    ▼
   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
   │  AgentVisor #1  │  │  AgentVisor #2  │  │  AgentVisor #3  │
   │   (stateless)   │  │   (stateless)   │  │   (stateless)   │
   └────────┬────────┘  └────────┬────────┘  └────────┬────────┘
            │                    │                    │
            └────────────────────┼────────────────────┘
                                 ▼
                    ┌─────────────────────────┐
                    │    Temporal Cluster     │
                    │   (all state stored)    │
                    └─────────────────────────┘

Request Routing

When a request arrives at any AgentVisor node:

1. Node receives the HTTP request (e.g., POST /threads/{id}/runs)
2. Node queries Temporal for the workflow state
3. Node executes the agent activity
4. Temporal persists the result
5. Response returns to client

Because all state lives in Temporal, the request can be handled by any node in the cluster.

Horizontal Scaling Pattern

Multiple Instances Per Agent Deployment

Scale AgentVisor horizontally by running multiple instances:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agentvisor
spec:
  replicas: 3  # Scale horizontally
  selector:
    matchLabels:
      app: agentvisor
  template:
    metadata:
      labels:
        app: agentvisor
    spec:
      containers:
      - name: agent
        image: my-agent:v1
        securityContext:
          seccompProfile:
            type: Unconfined  # Required for rootless gVisor (default mode)
          appArmorProfile:
            type: Unconfined  # Required on Ubuntu 24.04+, GKE, AKS
        ports:
        - containerPort: 8090
        env:
        - name: AGENTVISOR_TEMPORAL_TARGET
          value: "temporal.temporal-system.svc:7233"
        - name: AGENTVISOR_TEMPORAL_NAMESPACE
          value: "my-agent-prod"
        - name: AGENTVISOR_TEMPORAL_TASK_QUEUE
          value: "my-agent-tasks"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2"

Load Balancing

Use a Kubernetes Service for load balancing:

apiVersion: v1
kind: Service
metadata:
  name: agentvisor
spec:
  selector:
    app: agentvisor
  ports:
  - port: 8090
    targetPort: 8090
  type: ClusterIP

No sticky sessions are required because AgentVisor instances are stateless.

Why This Works

The Temporal task queue model ensures correct execution:

1. All AgentVisor instances poll the same task queue
2. Temporal assigns each activity execution to one worker
3. State is stored in the workflow, not the worker
4. Activities are retried automatically on failure

High Availability Deployment

Minimum Replicas

For production high availability, run at least 3 replicas:

spec:
  replicas: 3

This ensures:

• Continued operation during rolling updates
• Tolerance for single-node failures
• Capacity for traffic spikes

Spread Across Availability Zones

Use topology spread constraints to distribute pods across zones:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agentvisor
spec:
  replicas: 3
  selector:
    matchLabels:
      app: agentvisor
  template:
    metadata:
      labels:
        app: agentvisor
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: agentvisor
      containers:
      - name: agent
        image: my-agent:v1
        securityContext:
          seccompProfile:
            type: Unconfined  # Required for rootless gVisor (default mode)
          appArmorProfile:
            type: Unconfined  # Required on Ubuntu 24.04+, GKE, AKS
        ports:
        - containerPort: 8090
        env:
        - name: AGENTVISOR_TEMPORAL_TARGET
          value: "temporal.temporal-system.svc:7233"
        - name: AGENTVISOR_GUEST_SANDBOX
          value: "gvisor"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2"

PodDisruptionBudget

Protect against voluntary disruptions (node drains, upgrades):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: agentvisor
spec:
  minAvailable: 2  # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: agentvisor

This ensures at least 2 pods remain available during:

• Kubernetes upgrades
• Node maintenance
• Voluntary pod evictions

Complete HA Manifest

Here's a complete high-availability deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agentvisor
  labels:
    app: agentvisor
spec:
  replicas: 3
  selector:
    matchLabels:
      app: agentvisor
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: agentvisor
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: agentvisor
      containers:
      - name: agent
        image: my-agent:v1
        securityContext:
          seccompProfile:
            type: Unconfined  # Required for rootless gVisor (default mode)
          appArmorProfile:
            type: Unconfined  # Required on Ubuntu 24.04+, GKE, AKS
        ports:
        - containerPort: 8090
          name: http
        env:
        - name: AGENTVISOR_TEMPORAL_TARGET
          value: "temporal.temporal-system.svc:7233"
        - name: AGENTVISOR_TEMPORAL_NAMESPACE
          value: "production"
        - name: AGENTVISOR_TEMPORAL_TASK_QUEUE
          value: "my-agent"
        - name: AGENTVISOR_GUEST_SANDBOX
          value: "gvisor"
        - name: AGENTVISOR_TELEMETRY_METRICS_ENABLED
          value: "true"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2"
        livenessProbe:
          httpGet:
            path: /health
            port: 8090
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8090
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: agentvisor
spec:
  selector:
    app: agentvisor
  ports:
  - name: http
    port: 8090
    targetPort: 8090
  type: ClusterIP
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: agentvisor
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: agentvisor

Auto-Scaling Configuration

HorizontalPodAutoscaler

Scale based on CPU and memory utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: agentvisor
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: agentvisor
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scaling down
      policies:
      - type: Pods
        value: 1
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0  # Scale up immediately
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60

Scaling Considerations

CPU vs Memory scaling:

• CPU: Scales based on agent execution load (LLM calls, tool processing)
• Memory: Scales based on concurrent agent instances in sandbox

Recommended thresholds:

• CPU: 70% utilization triggers scale-up
• Memory: 80% utilization triggers scale-up
• Scale down slowly (5-minute stabilization) to avoid thrashing

Maximum replicas:

• Consider your Temporal cluster capacity
• Each replica creates connections to Temporal
• Account for LLM API rate limits across all replicas

Resilience Characteristics

Node Failure Handling

When an AgentVisor node fails mid-execution, Temporal's own native activity retry is not what recovers the run — AgentVisor deliberately configures agent invocation activities as non-retriable at the native-retry level, because replaying a partially completed agent run against external side effects (API calls, tool use) is not generally safe. That much has not changed. What has changed is that AgentVisor's own workflow code now restarts the run itself, within bounds, for exactly this failure mode:

1. HTTP request is in flight — the client is waiting (or polling)
2. Temporal detects the missed heartbeat — the failed node's activity is marked failed once the heartbeat timeout elapses (60 seconds by default — this is the floor on failover latency, and what step 4's "no different from a slow run" claim rests on); this classifies as an infrastructure failure (see below), not an agent error
3. AgentVisor's workflow-level restart loop retries the run — a fresh InvokeRun activity is scheduled (any healthy node picks it up, not necessarily the one that failed), resuming from the run's last checkpoint, after an exponential backoff. This repeats up to temporal.run_restart's configured attempt budget (default 3 restarts) and wall-clock ceiling (default 10 minutes)
4. The client sees no difference from a slow run — if a synchronous wait//runs/wait call is in progress, the backoff and retries happen within its existing wait window (up to its own ceiling); if the restart budget is exhausted, the run fails as error exactly as it always did
5. Workflow state is preserved throughout — the thread's history and prior checkpoints survive in Temporal regardless of outcome

This restart behavior is bounded and does not apply to every failure: an agent-raised exception, the activity's own StartToClose time-budget timeout, and cancellation are still never retried — the client must start a new run against the same thread in those cases, as before. See Recovery Scenarios for a caller-facing summary of which failure modes restart automatically. The activity options are still fixed in AgentVisor's code, not user-configurable — only the restart loop wrapped around them is:

// internal/host/temporal/workflow/execution.go
workflow.ActivityOptions{
    StartToCloseTimeout: 30 * time.Minute, // Agents can run for a while
    HeartbeatTimeout:    60 * time.Second,
    WaitForCancellation: true, // Wait for the activity to actually stop before resolving cancel
    RetryPolicy: &temporal.RetryPolicy{
        MaximumAttempts: 1, // Native retry stays disabled -- see the workflow-level
                             // restart loop, gated on failure classification, instead.
    },
}

Network Partition Behavior

During network partitions:

• Requests to isolated nodes fail
• Load balancer routes to healthy nodes
• Temporal activities on isolated nodes miss their heartbeat and time out — classified as an infrastructure failure
• The affected run restarts automatically on a healthy node, bounded by temporal.run_restart's attempt budget and wall-clock ceiling (see Node Failure Handling above); if that budget is exhausted before the partition heals, the run fails and the client must re-issue
• Thread/workflow state up to the last checkpoint remains consistent in Temporal throughout

Graceful Shutdown

AgentVisor handles graceful shutdown in order:

1. Stops the HTTP API server, draining in-flight requests for up to a hardcoded 10-second timeout (not configurable)
2. Stops accepting new guest connections
3. Cooperatively shuts down the guest sandbox (signals the guest to drain its agents), waiting up to AGENTVISOR_GUEST_SHUTDOWN_TIMEOUT before force-closing
4. Closes the hostlink and gRPC servers, then the trace pipeline

AGENTVISOR_GUEST_SHUTDOWN_TIMEOUT only governs step 3 (guest sandbox teardown) — there is no separate knob for the HTTP request drain:

AGENTVISOR_GUEST_SHUTDOWN_TIMEOUT=30s

In Kubernetes, ensure terminationGracePeriodSeconds exceeds the sum of the fixed 10-second HTTP drain and AGENTVISOR_GUEST_SHUTDOWN_TIMEOUT:

spec:
  terminationGracePeriodSeconds: 60

Request Timeout Configuration

Configure timeouts at each layer:

# Guest sandbox startup
AGENTVISOR_GUEST_STARTUP_TIMEOUT=30s

# Guest sandbox shutdown
AGENTVISOR_GUEST_SHUTDOWN_TIMEOUT=10s

# Individual HTTP proxy requests
AGENTVISOR_PROXY_REQUEST_TIMEOUT=30s

Temporal Cluster Scaling

AgentVisor's scalability depends on Temporal's scalability.

Self-Hosted Temporal

For self-hosted Temporal clusters:

Database scaling:

• Use a production database (PostgreSQL, MySQL, Cassandra, Yugabyte)
• Scale database resources based on workflow volume
• Enable connection pooling

History service scaling:

• Scale history service horizontally for more concurrent workflows
• Each history shard handles a subset of workflows

Frontend service scaling:

• Scale frontend for more concurrent API connections
• AgentVisor instances connect to frontend

See Temporal Server documentation for detailed guidance.

Temporal Cloud

Temporal Cloud provides managed scaling:

• Automatic scaling — infrastructure scales with usage
• Global regions — deploy close to your users
• Namespace isolation — separate workloads per namespace

Namespace rate limits apply:

• Actions per second (APS) limits
• Concurrent workflow limits
• Request concurrency limits

Contact Temporal support to adjust limits for high-volume workloads.

Namespace Strategy

Use separate namespaces for:

• Different agents (e.g., chatbot-prod, research-prod)
• Different environments (e.g., myagent-dev, myagent-staging, myagent-prod)
• Different tenants in multi-tenant deployments

AGENTVISOR_TEMPORAL_NAMESPACE=myagent-prod
AGENTVISOR_TEMPORAL_TASK_QUEUE=myagent-tasks

Each agent deployment should use a unique namespace+task_queue combination to prevent workflow routing conflicts.

Resource Planning

Memory Requirements

Per concurrent agent execution:

• Base overhead: ~50-100 MB (guest runtime, gRPC connections)
• Python runtime: ~100-200 MB
• Agent code and dependencies: Varies (100 MB - 1 GB+)
• LangGraph state: Depends on conversation history

Example sizing:

• 5 concurrent agents × 400 MB each = 2 GB minimum
• Add 50% headroom = 3 GB recommended

CPU Considerations

Agent CPU usage depends on:

• LLM response processing
• Tool execution
• Python computation

Most agent workloads are I/O bound (waiting for LLM APIs), not CPU bound. Start with:

• 500m CPU request
• 2 CPU limit

Monitor actual usage and adjust.

Connection Pool Sizing

AgentVisor maintains connection pools for MCP servers and HTTP proxy connections.

MCP connection pool:

# Maximum connections in pool
AGENTVISOR_MCP_POOL_MAX_SIZE=100

# Close idle connections after this duration
AGENTVISOR_MCP_POOL_IDLE_TIMEOUT=5m

Sizing guidance:

• Set MAX_SIZE to expected peak concurrent MCP-using agents
• Connections are per-principal for principal-bound auth
• Monitor pool exhaustion in metrics

HTTP proxy connections:

• No configurable pool; connections are made per-request
• Each concurrent agent can have multiple in-flight HTTP requests
• Monitor connection counts in metrics

Recommended Starting Configuration

For a typical production deployment:

Replicas	Memory Request	Memory Limit	CPU Request	CPU Limit
3	512 Mi	2 Gi	500m	2

Adjust based on:

• Observed resource usage
• Agent complexity
• Concurrent execution requirements

Monitoring for Scale

Key Metrics to Watch

AgentVisor metrics:

• agentvisor_http_requests_total — Request rate and errors
• agentvisor_http_request_duration_seconds — Latency distribution
• agentvisor_runs_total — Agent run count (increment on start/completion)
• agentvisor_run_duration_seconds — Agent run duration distribution
• agentvisor_mcp_pool_connections — MCP pool utilization

Temporal metrics:

• temporal_workflow_execute_latency — Workflow execution time
• temporal_activity_execution_total — Activity execution count
• temporal_activity_task_schedule_to_start_latency — Queue wait time (indicates capacity issues)

Kubernetes metrics:

• Pod CPU and memory usage
• Pod restart count
• HPA scaling events

Prometheus/Grafana Setup

Enable metrics in AgentVisor. The /metrics endpoint is served on the same shared HTTP API listener as the rest of the API (AGENTVISOR_API_LISTEN_ADDR, default :8090) — there is no separate metrics bind address:

AGENTVISOR_TELEMETRY_METRICS_ENABLED=true

Create a ServiceMonitor for Prometheus:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: agentvisor
spec:
  selector:
    matchLabels:
      app: agentvisor
  endpoints:
  - port: http
    path: /metrics
    interval: 30s

Alerting Recommendations

Set up alerts for:

Condition	Threshold	Severity
High error rate	>5% 5xx responses over 5m	Warning
High latency	p99 > 30s over 5m	Warning
Pod restarts	>3 in 10m	Critical
HPA at max replicas	maxReplicas reached for 10m	Warning
Temporal task queue backlog	Schedule-to-start latency > 5s	Warning
Memory near limit	>90% for 5m	Warning

Example Prometheus alert:

groups:
- name: agentvisor
  rules:
  - alert: AgentVisorHighErrorRate
    expr: |
      sum(rate(agentvisor_http_requests_total{status=~"5.."}[5m])) /
      sum(rate(agentvisor_http_requests_total[5m])) > 0.05
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "AgentVisor error rate exceeds 5%"
      description: "Error rate is {{ $value | humanizePercentage }}"

Summary

Key takeaways for scaling AgentVisor:

1. Stateless architecture — All state in Temporal, any node handles any request
2. Horizontal scaling — Add replicas to increase capacity
3. High availability — 3+ replicas across availability zones with PDB
4. Auto-scaling — HPA based on CPU/memory with conservative scale-down
5. Temporal is the bottleneck — Ensure your Temporal cluster can handle the load
6. Monitor thoroughly — Track AgentVisor, Temporal, and Kubernetes metrics