Scaling and Resilience

This guide covers how to scale AgentVisor™ deployments and ensure high availability for production workloads.

Architecture Foundation

AgentVisor achieves resilience through its integration with Temporal. Understanding this architecture is key to scaling effectively.

Temporal as the Resilience Backbone

All conversation state (threads, runs, checkpoints) is stored in Temporal workflows, not in AgentVisor instances. This means:

• Stateless nodes: AgentVisor instances hold no persistent state
• Any node can handle any request: Load balancers don't need sticky sessions
• Durable workflow state: If a node dies mid-request, the thread's workflow state survives in Temporal and a new node can pick up the next request — the in-flight run itself restarts automatically, bounded, instead of failing outright (see Node Failure Handling below)
• Workflow affinity: Temporal handles routing to the correct workflow context

┌─────────────────────────────────────────────────────────────────┐
│                        Load Balancer                            │
└─────────────────────────────────────────────────────────────────┘
                                 │
            ┌────────────────────┼────────────────────┐
            ▼                    ▼                    ▼
   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
   │  AgentVisor #1  │  │  AgentVisor #2  │  │  AgentVisor #3  │
   │   (stateless)   │  │   (stateless)   │  │   (stateless)   │
   └────────┬────────┘  └────────┬────────┘  └────────┬────────┘
            │                    │                    │
            └────────────────────┼────────────────────┘
                                 ▼
                    ┌─────────────────────────┐
                    │    Temporal Cluster     │
                    │   (all state stored)    │
                    └─────────────────────────┘

Request Routing

When a request arrives at any AgentVisor node:

1. Node receives the HTTP request (e.g., POST /threads/{id}/runs)
2. Node queries Temporal for the workflow state
3. Node executes the agent activity
4. Temporal persists the result
5. Response returns to client

Because all state lives in Temporal, the request can be handled by any node in the cluster.

Horizontal Scaling Pattern

Multiple Instances Per Agent Deployment

Scale AgentVisor horizontally by running multiple instances:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agentvisor
spec:
  replicas: 3  # Scale horizontally
  selector:
    matchLabels:
      app: agentvisor
  template:
    metadata:
      labels:
        app: agentvisor
    spec:
      containers:
      - name: agent
        image: my-agent:v1
        securityContext:
          seccompProfile:
            type: Unconfined  # Required for rootless gVisor (default mode)
          appArmorProfile:
            type: Unconfined  # Required on Ubuntu 24.04+, GKE, AKS
        ports:
        - containerPort: 8090
        env:
        - name: AGENTVISOR_TEMPORAL_TARGET
          value: "temporal.temporal-system.svc:7233"
        - name: AGENTVISOR_TEMPORAL_NAMESPACE
          value: "my-agent-prod"
        - name: AGENTVISOR_TEMPORAL_TASK_QUEUE
          value: "my-agent-tasks"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2"

Load Balancing

Use a Kubernetes Service for load balancing:

apiVersion: v1
kind: Service
metadata:
  name: agentvisor
spec:
  selector:
    app: agentvisor
  ports:
  - port: 8090
    targetPort: 8090
  type: ClusterIP

No sticky sessions are required because AgentVisor instances are stateless.

Why This Works

The Temporal task queue model ensures correct execution:

1. All AgentVisor instances poll the same task queue
2. Temporal assigns each activity execution to one worker
3. State is stored in the workflow, not the worker
4. Activities are retried automatically on failure

High Availability Deployment

Minimum Replicas

For production high availability, run at least 3 replicas:

spec:
  replicas: 3

This ensures:

• Continued operation during rolling updates
• Tolerance for single-node failures
• Capacity for traffic spikes

Spread Across Availability Zones

Use topology spread constraints to distribute pods across zones:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agentvisor
spec:
  replicas: 3
  selector:
    matchLabels:
      app: agentvisor
  template:
    metadata:
      labels:
        app: agentvisor
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: agentvisor
      containers:
      - name: agent
        image: my-agent:v1
        securityContext:
          seccompProfile:
            type: Unconfined  # Required for rootless gVisor (default mode)
          appArmorProfile:
            type: Unconfined  # Required on Ubuntu 24.04+, GKE, AKS
        ports:
        - containerPort: 8090
        env:
        - name: AGENTVISOR_TEMPORAL_TARGET
          value: "temporal.temporal-system.svc:7233"
        - name: AGENTVISOR_GUEST_SANDBOX
          value: "gvisor"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2"

PodDisruptionBudget

Protect against voluntary disruptions (node drains, upgrades). With the recommended 3 replicas, minAvailable: 2 allows exactly one pod to be evicted at a time — enough to keep a rolling update or node drain moving without dropping below 2 healthy pods:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: agentvisor
spec:
  minAvailable: 2  # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: agentvisor

This ensures at least 2 pods remain available during:

• Kubernetes upgrades
• Node maintenance
• Voluntary pod evictions

Complete HA Manifest

Here's a complete high-availability deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agentvisor
  labels:
    app: agentvisor
spec:
  replicas: 3
  selector:
    matchLabels:
      app: agentvisor
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: agentvisor
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: agentvisor
      containers:
      - name: agent
        image: my-agent:v1
        securityContext:
          seccompProfile:
            type: Unconfined  # Required for rootless gVisor (default mode)
          appArmorProfile:
            type: Unconfined  # Required on Ubuntu 24.04+, GKE, AKS
        ports:
        - containerPort: 8090
          name: http
        env:
        - name: AGENTVISOR_TEMPORAL_TARGET
          value: "temporal.temporal-system.svc:7233"
        - name: AGENTVISOR_TEMPORAL_NAMESPACE
          value: "production"
        - name: AGENTVISOR_TEMPORAL_TASK_QUEUE
          value: "my-agent"
        - name: AGENTVISOR_GUEST_SANDBOX
          value: "gvisor"
        - name: AGENTVISOR_TELEMETRY_METRICS_ENABLED
          value: "true"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2"
        livenessProbe:
          httpGet:
            path: /health
            port: 8090
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8090
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: agentvisor
spec:
  selector:
    app: agentvisor
  ports:
  - name: http
    port: 8090
    targetPort: 8090
  type: ClusterIP
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: agentvisor
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: agentvisor

Auto-Scaling Configuration

HorizontalPodAutoscaler

Scale based on CPU and memory utilization. minReplicas matches the minimum replica count recommended for high availability, so autoscaling never scales below the HA floor:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: agentvisor
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: agentvisor
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scaling down
      policies:
      - type: Pods
        value: 1
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0  # Scale up immediately
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60

Scaling Considerations

CPU vs Memory scaling:

• CPU: Scales based on agent execution load (LLM calls, tool processing)
• Memory: Scales based on concurrent agent instances in sandbox

Recommended thresholds:

• CPU: 70% utilization triggers scale-up
• Memory: 80% utilization triggers scale-up
• Scale down slowly (5-minute stabilization) to avoid thrashing

Maximum replicas:

• Consider your Temporal cluster capacity
• Each replica creates connections to Temporal
• Account for LLM API rate limits across all replicas

Resilience Characteristics

Node Failure Handling

When an AgentVisor node fails mid-execution, Temporal's own native activity retry is not what recovers the run — AgentVisor deliberately configures agent invocation activities as non-retriable at the native-retry level, because replaying a partially completed agent run against external side effects (API calls, tool use) is not generally safe. Instead, AgentVisor's own workflow code restarts the run itself, within bounds, for exactly this failure mode:

1. HTTP request is in flight — the client is waiting (or polling)
2. Temporal detects the missed heartbeat — the failed node's activity is marked failed once the heartbeat timeout elapses (60 seconds by default — this is the floor on failover latency, and what step 4's "no different from a slow run" claim rests on); this classifies as an infrastructure failure (see below), not an agent error
3. AgentVisor's workflow-level restart loop retries the run — a fresh invocation is scheduled (any healthy node picks it up, not necessarily the one that failed), resuming from the run's last checkpoint, after an exponential backoff. This repeats up to temporal.run_restart's configured attempt budget (default 3 restarts) and wall-clock ceiling (default 10 minutes)
4. The client sees no difference from a slow run — if a synchronous wait//runs/wait call is in progress, the backoff and retries happen within its existing wait window (up to its own ceiling); if the restart budget is exhausted, the run fails as error
5. Workflow state is preserved throughout — the thread's history and prior checkpoints survive in Temporal regardless of outcome

This restart behavior is bounded and does not apply to every failure: an agent-raised exception, the activity's own StartToClose time-budget timeout, and cancellation are never retried — the client must start a new run against the same thread in those cases. See Recovery Scenarios for a caller-facing summary of which failure modes restart automatically. The activity options below are fixed, not user-configurable — only the restart loop wrapped around them is:

Setting	Value	Notes
Start-to-close timeout	30 minutes	Agents can run for a while
Heartbeat timeout	60 seconds	Also the basis for the ~48-second cancellation-fallback floor below
Wait for cancellation	Enabled	Waits for the activity to actually stop before resolving a cancel
Native activity retry	Disabled (max 1 attempt)	The restart loop above replaces it, gated on failure classification

Run Cancellation Latency

Temporal only ever delivers activity cancellation to a worker via a heartbeat RPC response — there is no other delivery channel. With the 60-second heartbeat timeout above, that floors plain context-cancellation delivery at roughly 48 seconds (0.8 * HeartbeatTimeout, the interval at which the Temporal SDK batches heartbeat delivery), regardless of how quickly the cancel request itself reaches the workflow.

AgentVisor closes most of that gap with a fast path, additive to that fallback rather than a replacement for it:

1. A run's agent invocation is pinned to a worker-scoped Temporal Session for the life of its attempt loop — the same session that also backs the automatic-restart loop above.
2. A cancel request dispatches a second, small activity within that same session — Temporal guarantees a session-scoped activity lands on the exact worker process already running the target agent invocation. Scheduling a new activity is normal- latency Temporal work (typically sub-second on a healthy cluster), unlike interrupting an activity already in flight — which is what the heartbeat floor above actually gates.
3. That activity reaches into the worker's own in-memory execution state and cancels the agent process directly — the same cleanup the heartbeat-driven fallback path eventually performs, just without waiting on the heartbeat throttle.
4. If the fast path is unavailable for any reason — session creation failed, the session's own worker died, or the fast-path activity itself errored or found nothing to cancel — cancellation still lands via the fallback path above, with its unchanged ~48-second floor. The fast path never makes cancellation less reliable, only faster in the common case.

Expect low single-digit-second cancellation latency in normal operation, with the ~48-second fallback floor as the worst case an operator should plan around — see Run Cancellation as a Containment Control for why that worst case matters when cancellation is used to contain a runaway or compromised agent rather than just to stop a run a caller no longer wants. agentvisor_run_cancel_fast_path_total and agentvisor_run_cancel_fallback_total (tagged trigger=cancel for an explicit run cancellation, plus fallback reason) show which path actually delivered a cancellation — see Monitoring for Scale below for the full metric reference, including the continue_as_new trigger these same counters also carry for a thread's own internal history-management activity. temporal.run_cancellation tunes this mechanism's session heartbeat timeout and per-worker session capacity — see Fast Run Cancellation for the full config reference.

Session Capacity

max_concurrent_sessions bounds how many runs can have fast-path cancellation active on one worker at once — one session is held open per actively-executing run's attempt loop, for its entire duration. A worker at that limit still cancels every additional concurrent run reliably; those runs simply use the ordinary fallback path instead of the fast one. Size this to the number of concurrent runs you expect a single worker to host, not against any other worker capacity setting.

Network Partition Behavior

During network partitions:

• Requests to isolated nodes fail
• Load balancer routes to healthy nodes
• Temporal activities on isolated nodes miss their heartbeat and time out — classified as an infrastructure failure
• The affected run restarts automatically on a healthy node, bounded by temporal.run_restart's attempt budget and wall-clock ceiling (see Node Failure Handling above); if that budget is exhausted before the partition heals, the run fails and the client must re-issue
• Thread/workflow state up to the last checkpoint remains consistent in Temporal throughout

Graceful Shutdown

AgentVisor handles graceful shutdown in order:

1. Stops the HTTP API server, draining in-flight requests for up to a hardcoded 10-second timeout (not configurable)
2. Stops accepting new guest connections
3. Cooperatively shuts down the guest sandbox (signals the guest to drain its agents), waiting up to AGENTVISOR_GUEST_SHUTDOWN_TIMEOUT before force-closing
4. Closes the hostlink and gRPC servers, then the trace pipeline

AGENTVISOR_GUEST_SHUTDOWN_TIMEOUT only governs step 3 (guest sandbox teardown) — there is no separate knob for the HTTP request drain:

AGENTVISOR_GUEST_SHUTDOWN_TIMEOUT=30s

In Kubernetes, ensure terminationGracePeriodSeconds exceeds the sum of the fixed 10-second HTTP drain and AGENTVISOR_GUEST_SHUTDOWN_TIMEOUT:

spec:
  terminationGracePeriodSeconds: 60

Request Timeout Configuration

Configure timeouts at each layer:

# Guest sandbox startup
AGENTVISOR_GUEST_STARTUP_TIMEOUT=30s

# Guest sandbox shutdown
AGENTVISOR_GUEST_SHUTDOWN_TIMEOUT=10s

# Individual HTTP proxy requests
AGENTVISOR_PROXY_REQUEST_TIMEOUT=30s

Temporal Cluster Scaling

AgentVisor's scalability depends on Temporal's scalability.

Self-Hosted Temporal

For self-hosted Temporal clusters:

Database scaling:

• Use a production database (PostgreSQL, MySQL, Cassandra, Yugabyte)
• Scale database resources based on workflow volume
• Enable connection pooling

History service scaling:

• Scale history service horizontally for more concurrent workflows
• Each history shard handles a subset of workflows

Frontend service scaling:

• Scale frontend for more concurrent API connections
• AgentVisor instances connect to frontend

See Temporal Server documentation for detailed guidance.

Temporal Cloud

Temporal Cloud provides managed scaling:

• Automatic scaling — infrastructure scales with usage
• Global regions — deploy close to your users
• Namespace isolation — separate workloads per namespace

Namespace rate limits apply:

• Actions per second (APS) limits
• Concurrent workflow limits
• Request concurrency limits

Contact Temporal support to adjust limits for high-volume workloads.

Namespace Strategy

Use separate namespaces for:

• Different agents (e.g., chatbot-prod, research-prod)
• Different environments (e.g., myagent-dev, myagent-staging, myagent-prod)
• Different tenants in multi-tenant deployments

AGENTVISOR_TEMPORAL_NAMESPACE=myagent-prod
AGENTVISOR_TEMPORAL_TASK_QUEUE=myagent-tasks

Each agent deployment should use a unique namespace+task_queue combination to prevent workflow routing conflicts.

Resource Planning

Memory Requirements

Per concurrent agent execution:

• Base overhead: ~50-100 MB (guest runtime, gRPC connections)
• Python runtime: ~100-200 MB
• Agent code and dependencies: Varies (100 MB - 1 GB+)
• LangGraph state: Depends on conversation history

Example sizing:

• 5 concurrent agents × 400 MB each = 2 GB minimum
• Add 50% headroom = 3 GB recommended

CPU Considerations

Agent CPU usage depends on:

• LLM response processing
• Tool execution
• Python computation

Most agent workloads are I/O bound (waiting for LLM APIs), not CPU bound. Start with:

• 500m CPU request
• 2 CPU limit

Monitor actual usage and adjust.

Connection Pool Sizing

AgentVisor maintains connection pools for MCP servers and HTTP proxy connections.

MCP connection pool:

# Maximum connections in pool
AGENTVISOR_MCP_POOL_MAX_SIZE=100

# Close idle connections after this duration
AGENTVISOR_MCP_POOL_IDLE_TIMEOUT=5m

Sizing guidance:

• Set MAX_SIZE to expected peak concurrent MCP-using agents
• Connections are per-principal for principal-bound auth
• Monitor pool exhaustion in metrics

HTTP proxy connections:

• No configurable pool; connections are made per-request
• Each concurrent agent can have multiple in-flight HTTP requests
• Monitor connection counts in metrics

Recommended Starting Configuration

For a typical production deployment:

Replicas	Memory Request	Memory Limit	CPU Request	CPU Limit
3	512 Mi	2 Gi	500m	2

Adjust based on:

• Observed resource usage
• Agent complexity
• Concurrent execution requirements

Monitoring for Scale

Key Metrics to Watch

AgentVisor metrics:

• agentvisor_http_requests_total — Request rate and errors
• agentvisor_http_request_duration_seconds — Latency distribution
• agentvisor_runs_total — Agent run count (increment on start/completion)
• agentvisor_run_duration_seconds — Agent run duration distribution
• agentvisor_mcp_pool_connections — MCP pool utilization

Temporal metrics:

• temporal_workflow_execute_latency — Workflow execution time
• temporal_activity_execution_total — Activity execution count
• temporal_activity_task_schedule_to_start_latency — Queue wait time (indicates capacity issues)

Kubernetes metrics:

• Pod CPU and memory usage
• Pod restart count
• HPA scaling events

Prometheus/Grafana Setup

Enable metrics in AgentVisor. The /metrics endpoint is served on the same shared HTTP API listener as the rest of the API (AGENTVISOR_API_LISTEN_ADDR, default :8090) — there is no separate metrics bind address:

AGENTVISOR_TELEMETRY_METRICS_ENABLED=true

Create a ServiceMonitor for Prometheus:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: agentvisor
spec:
  selector:
    matchLabels:
      app: agentvisor
  endpoints:
  - port: http
    path: /metrics
    interval: 30s

Alerting Recommendations

Set up alerts for:

Condition	Threshold	Severity
High error rate	>5% 5xx responses over 5m	Warning
High latency	p99 > 30s over 5m	Warning
Pod restarts	>3 in 10m	Critical
HPA at max replicas	maxReplicas reached for 10m	Warning
Temporal task queue backlog	Schedule-to-start latency > 5s	Warning
Memory near limit	>90% for 5m	Warning

Example Prometheus alert:

groups:
- name: agentvisor
  rules:
  - alert: AgentVisorHighErrorRate
    expr: |
      sum(rate(agentvisor_http_requests_total{status=~"5.."}[5m])) /
      sum(rate(agentvisor_http_requests_total[5m])) > 0.05
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "AgentVisor error rate exceeds 5%"
      description: "Error rate is {{ $value | humanizePercentage }}"

Summary

Key takeaways for scaling AgentVisor:

1. Stateless architecture — All state in Temporal, any node handles any request
2. Horizontal scaling — Add replicas to increase capacity
3. High availability — 3+ replicas across availability zones with PDB
4. Auto-scaling — HPA based on CPU/memory with conservative scale-down
5. Temporal is the bottleneck — Ensure your Temporal cluster can handle the load
6. Monitor thoroughly — Track AgentVisor, Temporal, and Kubernetes metrics