Skip to main content

External Checkpoint Storage

By default, AgentVisor stores every checkpoint, pending-write batch, and compaction payload as part of Temporal's own workflow state. That needs no extra infrastructure — it's the zero-dependency default — but it caps each of those at roughly 2 MiB, Temporal's own per-payload limit.

External Checkpoint Storage raises that cap. Turn it on, point it at a backend you supply (PostgreSQL, S3, YugabyteDB, or the local filesystem), and any checkpoint, pending-write batch, or compaction payload that grows past a configurable size is stored there instead — at the cost of operating that extra storage layer. It's opt-in and off by default.

Do you need it?​

Only if a thread's checkpoints genuinely need to exceed a few MiB — most workloads never approach the default 2 MiB ceiling. If you're not sure whether you've hit it, see Checkpoint Sizing and Limits for how to tell, and treat this feature as something to reach for once you have, not something to enable reflexively for every deployment.

Enabling it​

temporal:
  # Required whenever checkpoint_storage.enabled is true: with large payloads
  # moved to the backend, workflow history shrinks, so Temporal's own history-size
  # trigger for this workflow's Continue-As-New fires much later than it otherwise
  # would -- this caps put_writes Updates per execution as a backstop.
  max_put_writes_per_execution: 500

  checkpoint_storage:
    enabled: true
    threshold: 1048576   # 1MiB (default) -- payloads at or above this size move to the backend
    provider: postgresql # memory | filesystem | postgresql | s3 | ycql
    postgresql:
      connection_string: "postgres://user:pass@localhost:5432/agentvisor"
    sweep:
      enabled: true   # default
      interval: 24h   # default
      grace_period: 24h # default

Or via environment variables:

export AGENTVISOR_TEMPORAL_MAX_PUT_WRITES_PER_EXECUTION=500
export AGENTVISOR_TEMPORAL_CHECKPOINT_STORAGE_ENABLED=true
export AGENTVISOR_TEMPORAL_CHECKPOINT_STORAGE_PROVIDER=postgresql
export AGENTVISOR_TEMPORAL_CHECKPOINT_STORAGE_POSTGRESQL_CONNECTION_STRING=postgres://user:pass@localhost:5432/agentvisor

See Configuration Reference: External Checkpoint Storage for the complete field/env-var list, including every provider's options.

Enabling it also raises the automatic temporal.max_checkpoint_bytes from 1 MiB to 8 MiB; set that field explicitly if you need a different ceiling. See Checkpoint Sizing and Limits: Raising the Ceiling for the full picture of what that knob governs.

Choosing a provider​

ProviderPersistenceUse case
memoryNone — lost on process restartThrowaway testing only
filesystemLocal disk (dir, default <cache_dir>/blobstore)Single-host evaluation or development — see Trying it out below
postgresqlDurable, sharedProduction — reuses the same TLS options and connection-pool tuning as the PostgreSQL store provider
s3Durable, sharedProduction — Amazon S3 or an S3-compatible service (e.g. MinIO), including a self-hosted one via endpoint
ycqlDurable, sharedProduction — YugabyteDB over its Cassandra-compatible YCQL protocol; supports an opt-in per-row TTL as a backstop alongside the cleanup mechanism (not a substitute for it)

ycql distributes its backend index across a fixed number of hash buckets (buckets, default 16); raise it for a deployment expecting a large workflow population — see Configuration Reference: External Checkpoint Storage for the exact tradeoff and its migration hazard.

All five backends implement the same interface and are exercised by a shared conformance test suite, so switching providers is a configuration change, not a code change. One exception: once a deployment has written any object under a given provider, don't switch providers for that deployment without a migration plan — there is no built-in cross-backend migration tool.

Trying it out​

The filesystem provider is the quickest way to evaluate this feature without provisioning any external infrastructure: it's durable across restarts, needs only a local directory, and works the same way in a single-host development setup as it does in a small production deployment.

export AGENTVISOR_TEMPORAL_CHECKPOINT_STORAGE_ENABLED=true
export AGENTVISOR_TEMPORAL_CHECKPOINT_STORAGE_PROVIDER=filesystem
export AGENTVISOR_TEMPORAL_CHECKPOINT_STORAGE_FILESYSTEM_DIR=/var/lib/agentvisor/blobstore

memory is the same idea but with nothing written to disk — useful for short-lived tests, not for anything you want to survive a restart. postgresql, s3, and ycql each work against any compatible service you already run or can stand up yourself (a self-hosted PostgreSQL/YugabyteDB instance, or any S3-compatible object store via endpoint) — see Configuration Reference: External Checkpoint Storage for each provider's connection options.

Sizing and growth​

Objects accumulate in proportion to how often threads write checkpoints and how often they compact — and, for a thread whose payloads sit above threshold, in proportion to read volume too: a read of a large checkpoint stores a fresh object in the backend on every call, same as a write, so a thread whose head checkpoint is read back once per invocation (LangGraph's own runner does exactly this) roughly doubles its object count over writes alone. Size threshold and expected backend load with that in mind, not just write rate.

Lowering threshold below its 1 MiB default only pays off once you've also raised temporal.max_checkpoint_bytes — otherwise a payload that already writes inline today gains nothing from being moved to the backend early, and only adds load on the backend and more objects to eventually clean up. threshold has its own independent upper bound (1.5 MiB) regardless of how high max_checkpoint_bytes goes: it names the inline/offload boundary, not the write-time ceiling, and a payload at or below threshold always stays inline in Temporal workflow history — where Temporal's raw per-payload ceiling still binds — so threshold must never scale up together with max_checkpoint_bytes.

Cleanup​

An object becomes eligible for cleanup once its owning thread is done and enough time has passed for it to safely leave Temporal's retention window. Reclamation is mostly self-scheduled: closing a thread (via compaction, cancellation, or ephemeral run completion) automatically schedules a delayed cleanup of that thread's own objects, timed to fire once the thread has left retention. A daily safety-net sweep covers the rest — an abnormal termination (a crash, an operator-initiated termination, a replay failure) that never got a chance to self-schedule its own cleanup. Neither ever touches anything younger than grace_period (default 24h) regardless of what it finds.

temporal:
  checkpoint_storage:
    sweep:
      enabled: true
      interval: 24h          # how often the safety-net sweep runs
      grace_period: 24h      # minimum object age before it's eligible for reclamation
      batch_size: 100        # keys handled per sweep scan/delete round trip
      call_timeout: 10s      # timeout for each liveness check a sweep makes
      concurrency: 10        # max liveness checks one sweep batch has in flight at once
      rate_limit: 20         # max liveness checks per second, across every batch
      live_cache_size: 4096  # max distinct executions one sweep batch dedupes via LRU
    cleanup:
      retention_margin: 1h  # safety buffer added to namespace retention before
                             # a per-thread cleanup re-checks liveness

A few things to plan around:

  • The safety-net sweep bounds its own load on Temporal. A sweep batch's liveness checks are concurrency-bounded (concurrency) and rate-limited (rate_limit), so a batch spanning many workflow executions can't alone spike load on Temporal — this matters most for a deployment with a large abnormal-termination residual population (the safety net exists for what the per-thread cleanup mechanism couldn't reach).
  • Object keys embed thread identity (namespace, workflow ID, run ID) in the backend's own storage — table rows, S3 keys and access logs, YCQL partitions. If the Temporal payload codec is enabled, the stored content is ciphertext, but the key itself is not.
  • Running multiple host replicas never means duplicated cleanup work. The safety-net sweep runs on a schedule that guarantees exactly one execution cluster-wide regardless of replica count; reclamation is idempotent per object regardless.
  • Once enabled and used, don't disable it. A thread whose history already references an object in the backend needs External Checkpoint Storage configured to replay at all — disabling it after any thread has stored a payload there risks breaking replay for that thread permanently, not just orphaning storage. Treat enabling this feature as a one-way decision for a given deployment. Startup enforces this: flipping enabled back to false while provider is still set is rejected by default (see Disabling external checkpoint storage for the break-glass override).

Metrics: agentvisor_checkpoint_storage_cleanup_objects_scanned_total and agentvisor_checkpoint_storage_cleanup_objects_reclaimed_total are recorded by both per-thread reclamation and the safety-net sweep. agentvisor_checkpoint_storage_cleanup_objects_skipped_total (tagged reason: live or liveness_check_error) is recorded only by the safety-net sweep — per-thread reclamation already confirms its target is dead before scanning, so it never has anything to skip. See Observability for how AgentVisor exposes OpenTelemetry metrics.

Operating it​

Every backend call is guarded by a per-call timeout (call_timeout, default 10s) and a circuit breaker (circuit_breaker.failure_threshold/circuit_breaker.reset_timeout, default: opens after 5 failures, resets after 30s), so a slow or unreachable backend fails fast instead of hanging workflow-task processing:

temporal:
  checkpoint_storage:
    call_timeout: 10s
    circuit_breaker:
      failure_threshold: 5
      reset_timeout: 30s

Watch agentvisor_checkpoint_storage_operations_total, agentvisor_checkpoint_storage_operation_duration_seconds, agentvisor_checkpoint_storage_digest_mismatch_total, agentvisor_checkpoint_storage_circuit_breaker_trips_total, and agentvisor_checkpoint_storage_circuit_breaker_state for backend health. Every retrieved payload's digest is independently recomputed and checked before use. See Checkpoint Sizing and Limits: How to tell you're near a limit for how these fit alongside AgentVisor's other size-limit signals.

Multiple deployments can safely share one backend. AgentVisor scopes every object to its own deployment automatically, and every provider enforces that scope structurally — scanning filters by it at the backend level (a WHERE clause, an S3 key prefix, a filesystem subdirectory), and a stored object's scope is checked before use. A deployment's cleanup can't enumerate, and a deployment can't read, another deployment's objects on a shared backend, even by accident.

See Also​