Skip to main content

Checkpoint Sizing and Limits

Checkpoints, pending writes, and compaction payloads all have to fit inside real size limits enforced by Temporal, the durable storage layer behind Checkpointing. This page is the reference for those limits, the knobs that raise them, and how to tell when a thread is approaching one.

Most workloads never get close to any of these ceilings. Treat this page as something to reach for once you suspect you're near a limit, not as required reading before shipping an agent.

Payload Size Limits​

Every checkpoint write, pending-write batch, and Continue-As-New (CAN) compaction is recorded as a single payload in Temporal's own workflow history, so all three are bound by Temporal's per-payload size limit by default — not by AgentVisor's own write-time size knob (temporal.max_checkpoint_bytes), which only bounds what AgentVisor itself is willing to attempt to write. Startup validation keeps that knob from being configured larger than Temporal's real ceiling can support unless External Checkpoint Storage is enabled to move the overflow out of workflow history — see Raising the Ceiling below. External Checkpoint Storage is the one mechanism that moves a payload out of this ceiling entirely; everything else on this page describes the limits that apply whether or not it's enabled.

LimitDefaultEffect
Per-payload size2 MiB, both on Temporal Cloud (fixed) and self-hosted Temporal (per-namespace configurable)The call fails, and a thread that hits this cannot make progress on that operation without a code fix or manual intervention. Treat 2 MiB as the number to budget against, with headroom for serialization framing overhead rather than the literal byte count — it is the tightest bound here, and the one every checkpoint write, pending-write batch, and CAN compaction is independently measured against.
Workflow history total size50 MiB error, with warnings starting well below thatThe workflow execution is terminated if it's ever actually reached. Approaching it is what triggers this workflow's own Continue-As-New automatically, well before the hard limit.
gRPC message size (Temporal worker ↔ server)4 MiBCaps an entire Workflow Task response (all commands combined) — a separate, larger ceiling than the 2 MiB per-payload limit above.
gRPC message size (guest ↔ host hostlink)50 MiB by default (AGENTVISOR_GUEST_CHECKPOINT_MAX_SIZE, set on the host process's environment)Bounds a SaveCheckpoint/LoadCheckpoint call over the guest↔host connection — a third, independent channel from the one above. The host resolves this once from its own environment and forwards the resolved value to the guest, so it sizes the host's own server, the guest's own server, and the guest's outbound dial identically, under every sandbox mode.

These are Temporal server defaults — AgentVisor does not override them. The 2 MiB per-payload limit is a per-namespace setting on self-hosted Temporal, so an operator running their own Temporal cluster can lower (or raise) it; it is fixed on Temporal Cloud.

How to tell you're near a limit​

A thread that's actually over a limit fails loudly, not silently — but the signal shows up in different places depending on which limit it hit:

  • A checkpoint or pending-write call is rejected before it even reaches Temporal. AgentVisor's own write-time validation, derived from the max_checkpoint_bytes ceiling below, is deliberately tighter than Temporal's raw 2 MiB, so this is usually the first thing you'll see, and it names the payload that was too large. Every such rejection also increments agentvisor_size_limit_rejections_total (tagged limit/operation) regardless of the guest's configured log level, so an operator can alert on the rate without reading logs at all — see Observability → Available Metrics.
  • A run ends with a CANPayloadBudgetExceeded error. This means a Continue-As-New compaction couldn't fit the thread's state under budget even after shedding everything it safely can — see Continue-As-New Budget below. Alert on the agentvisor_can_payload_budget_exceeded_total metric: it fires only in this case, and it means the thread's execution has stopped, not merely degraded.
  • Watch the CAN shedding metrics before they escalate that far. agentvisor_can_pending_writes_retained_bytes_total and agentvisor_can_messages_values_shed_total (see Observability → Available Metrics) tell you a thread is regularly consuming most of its CAN budget well before it ever fails outright.
  • A read — not just a write — can also fail once a checkpoint has grown too large. GET /threads/{thread_id}/state and the checkpoint-list/get queries can fail with a classified error when Temporal's own Query response payload exceeds its size ceiling (TMPRL1103). This is tracked separately from every other signal on this page, via agentvisor_query_payload_limit_exceeded_total (tagged query). The API response itself only reports that the thread's state is too large to read back — the specific fix for your deployment's own checkpoint storage configuration is logged for the operator, not returned to the caller.
  • Check workflow history size directly in the Temporal Web UI (http://localhost:8233 for a local dev stack) — a thread's "History" tab shows total event count and size, which correlates with how close it is to the workflow-history-size limit above.
  • GET /threads/{thread_id}/state returns the current checkpoint; its response size is a rough proxy for how large that checkpoint's payload is, without needing Temporal UI access.
  • If you're using External Checkpoint Storage, agentvisor_checkpoint_storage_payload_bytes (a histogram, tagged operation) shows how large the payloads it's offloading actually are — a proactive way to see usage trending toward a backend-side ceiling before anything actually fails.

Raising the Ceiling​

One config field governs what a single checkpoint (state + metadata + channel_values combined) is allowed to be: temporal.max_checkpoint_bytes, pinned into a thread's config at creation time rather than read live from the operator's running config on every call.

max_checkpoint_bytesMeaning
0 (default)Automatic — 1 MiB with External Checkpoint Storage disabled, 8 MiB with it enabled
explicit valueUsed as-is, up to a platform ceiling of 24 MiB

Everything else that scales with checkpoint size — the Continue-As-New payload budget, the total pending-writes bytes retained against a checkpoint, and the per-write value ceiling — derives automatically from max_checkpoint_bytes; you don't need to size those separately. See Configuration Reference: External Checkpoint Storage for the advanced override fields, if you ever need to set one independently.

Raising this value in operator config only benefits threads created after the change. An already-open thread never picks up a higher ceiling; there is no automatic migration path. If an existing thread needs a higher ceiling, create a new thread (POST /threads) rather than expecting a config change to retroactively apply to the old one.

Continue-As-New Budget​

When a Continue-As-New compaction fires, AgentVisor estimates the resulting payload against a budget derived from max_checkpoint_bytes and, if it doesn't fit, sheds state in a fixed order rather than failing immediately:

  1. Pending writes are dropped first. This costs the already-completed-task optimization for that specific compaction — sibling tasks in the same in-flight superstep may be re-executed after the automatic resume — but is otherwise harmless. In the common case of a small pending-write set, this doesn't happen at all.
  2. If that's still not enough, the run-output read model is shed. This is the data the REST/Agent Protocol API serves back to callers (not what LangGraph actually resumes execution from), so shedding it doesn't affect correctness.
  3. If the payload still doesn't fit, compaction fails loudly. The head checkpoint's own state is itself too large to shed safely — this is the CANPayloadBudgetExceeded case from How to tell you're near a limit above, and it ends the thread's execution rather than continuing it.

Because this shedding runs per Continue-As-New, a thread with many live checkpoint namespaces (heavy subgraph re-entry, or a wide fan-out step — see Subgraphs and Namespace Growth) hits the same sequence, since each namespace adds to the aggregate size CAN's estimate has to account for.

History Limits​

The workflow prunes older checkpoints to keep workflow state bounded:

# Maximum checkpoints retained per run (default: 1)
export AGENTVISOR_TEMPORAL_MAX_CHECKPOINT_HISTORY=1

The default of 1 keeps only the most recent checkpoint per run — enough for crash recovery, which only ever depends on the latest checkpoint (and its own pending writes, if a superstep is still in flight). Increase this if you need the LangGraph checkpointer's list() to return more than just the latest checkpoint, e.g. to inspect or revert to an earlier one; higher values retain more history at the cost of a proportionally larger Continue-As-New payload.

Retaining several checkpoints large enough to matter in practice generally requires raising max_checkpoint_bytes above and/or enabling External Checkpoint Storage below — otherwise the write-time ceiling caps what a single write can add to history regardless of how high this limit is set. A value above 1 also multiplies how much of the Continue-As-New payload budget the retained checkpoint history consumes, since each retained checkpoint counts toward the same estimate — the default max_checkpoint_bytes derivation assumes a single head checkpoint, so raise max_checkpoint_bytes (or the advanced can_payload_budget_bytes override) accordingly if you raise this.

This limit applies per checkpoint namespace, not once globally — see Subgraphs and Namespace Growth for how namespace count itself is bounded.

External Checkpoint Storage​

For payloads that need to exceed even a raised pinned ceiling — tens of megabytes rather than low single-digit MiB — AgentVisor can store a checkpoint, pending-write batch, or CAN payload at or above a configurable threshold in an operator-configured backend (PostgreSQL, S3, YugabyteDB YCQL, or the local filesystem) instead of inlining it in Temporal workflow history.

It's disabled by default (temporal.checkpoint_storage.enabled: false) — workflow-native storage stays the default and the only zero-dependency mode. Treat this as the mechanism for a thread whose checkpoints genuinely need to exceed a pinned per-thread ceiling, not as something to enable reflexively for every deployment. See the External Checkpoint Storage guide for how to enable and configure a backend, tune the reclamation sweep, and share one backend across multiple deployments.

See Also​