Checkpointing

AgentVisor™ provides durable state persistence for agents through Temporal workflow state, enabling crash recovery and human-in-the-loop patterns.

Overview

Checkpointing serves two purposes:

1. Durability: Agent state survives crashes and restarts
2. Human-in-the-Loop: Pause and resume agent execution

How It Works

LangGraph Integration

AgentVisorCheckpointer

The SDK provides a LangGraph-compatible checkpointer:

from agentvisor.langgraph import AgentVisorCheckpointer
from langgraph.graph import StateGraph

# Build your graph
builder = StateGraph(MyState)
builder.add_node("process", process_node)
builder.set_entry_point("process")

# Compile with AgentVisor checkpointer
checkpointer = AgentVisorCheckpointer()
graph = builder.compile(checkpointer=checkpointer)

# Thread ID is automatically read from AGENTVISOR_THREAD_ID
result = graph.invoke({"messages": [...]})

Automatic Checkpointing

LangGraph automatically checkpoints at each super-step (after all parallel nodes complete):

Checkpoint Data

The checkpointer saves:

• Graph state: The current TypedDict values
• Channel values: Accumulator states (like messages)
• Metadata: Timestamps, step counts, parent checkpoints

Pending writes (LangGraph's mid-superstep fault-tolerance mechanism, put_writes) are persisted too, on by default: each completed task's write within a superstep is saved as it happens, not just at superstep boundaries. If a crash interrupts an active run mid-superstep, replay restores already-completed tasks from their persisted writes instead of re-executing them — important for non-idempotent side effects (e.g. a tool call that isn't safe to repeat).

This protection has a size ceiling: an unusually large pending-write set for a single superstep may be dropped rather than carried through a compaction, at the cost of the already-completed-task optimization for that specific compaction (sibling tasks may be re-executed after the automatic resume — see Continue-As-New (CAN) below). See Checkpoint Sizing and Limits for exactly when this happens and what the fallback sequence looks like.

This is gated by temporal.pending_writes_enabled, which defaults to true but is pinned per thread at creation time: a thread created before this setting existed (or before it was enabled) has no such field recorded in its config and permanently resolves to disabled, for the life of that thread's Continue-As-New chain. See the AgentVisorCheckpointer reference for details, and Checkpoint Sizing and Limits for the config fields that bound pending-write size.

Subgraphs and Namespace Growth

LangGraph partitions checkpoint state by checkpoint_ns — the empty string for the root graph, and <parent_ns>|<node>:<task_id> for each subgraph invocation. AgentVisor's checkpoint store is namespace-aware: each namespace resolves its own head checkpoint independently, so a subgraph's get_tuple()/put() calls never read or overwrite its parent's (or a sibling subgraph's) checkpoint state. Each namespace's own history is bounded the same way as the root's — see History Limits.

Stale namespaces are reclaimed automatically, so a long-lived thread that repeatedly re-enters subgraphs doesn't accumulate namespaces without bound over its lifetime. A single wide fan-out superstep (Send/map-reduce over many items, each its own subgraph branch) can still cause a temporary spike, bounded by fan-out width times the root's retained history depth rather than the thread's entire lifetime. Disabling checkpoint pruning (MAX_CHECKPOINT_HISTORY <= 0) disables namespace reclamation too, since pruning is what triggers the check.

Copy Thread is a related limitation worth calling out here: a copy only ever duplicates the root namespace's current checkpoint, so copying a thread that has in-flight subgraph state does not carry that subgraph state over to the new thread.

Interrupt/Resume Pattern

Checkpointing enables human-in-the-loop patterns via LangGraph's interrupt() API:

from langgraph.types import interrupt

def human_review(state: AgentState) -> dict:
    """Pause for human review."""
    # Checkpoint is saved automatically before interrupt
    interrupt({
        "question": "Approve this action?",
        "action": state["pending_action"],
        "options": ["approve", "reject", "modify"]
    })
    # Execution stops here until resumed
    return {}

Resume Flow

Checkpointing outside LangGraph

For agents not built on LangGraph (custom Python, CrewAI crews, etc.), the SDK exposes a lower-level checkpointing API:

from agentvisor import checkpoint

# Save state
checkpoint_id = checkpoint.save("my-key", {"counter": 42, "items": ["a", "b"]})

# Load state
data = checkpoint.load(checkpoint_id)
print(data)  # {"counter": 42, "items": ["a", "b"]}

It durably persists any JSON-serializable state, keyed by name, using the same Temporal-backed workflow state as the LangGraph integration and surviving the same failure modes in Recovery Scenarios.

This API and the LangGraph checkpointer are not cross-compatible: checkpoint.load() only understands JSON state saved via checkpoint.save() and raises a clear ValueError (rather than a raw JSON decode error) if pointed at a checkpoint written by AgentVisorCheckpointer, which stores state as a typed LangGraph serde payload instead of JSON.

For Google ADK agents specifically, AgentVisor provides a session-scoped state mechanism via the Thread State API rather than the checkpoint API; see the ADK guide.

Strands agents map onto this API through Strands' own snapshot mechanism rather than a custom state shape: the AgentVisor Strands runner calls agent.take_snapshot(preset="session") — the "session" preset is Strands' own built-in field set (messages, state, conversation_manager_state, interrupt_state, model_state); take_snapshot() with no preset or include list raises SnapshotException — and passes .to_dict() of the result to checkpoint.save() after each run, then calls checkpoint.load("") (an empty ID means "latest checkpoint for this thread," since the runner is a fresh process on every invocation and has no saved ID to reuse from the prior run) and feeds the result through Snapshot.from_dict()/agent.load_snapshot() before the next run on the same thread — restoring conversation history and internal state automatically, with no code required in the agent itself. It carries the same caveat as any other consumer of this API: checkpoint.load() only understands JSON state saved via checkpoint.save(), so a Strands snapshot checkpoint is not cross-compatible with AgentVisorCheckpointer. Because interrupt_state round-trips through this same save/load path, an agent paused mid-run via Strands' own event.interrupt() resumes correctly on the very next invocation despite the runner being a fresh process each time — the restored agent already knows about its pending interrupt before that run's input is ever read.

A Graph or Swarm (MultiAgentBase) maps onto this API differently: the runner uses Strands' own serialize_state()/deserialize_state() pair instead of the snapshot API above, since take_snapshot()/load_snapshot() only exist on a plain Agent. This is narrower than the single-Agent case — it durably persists in-flight task and interrupt/HITL resume state, not cross-turn conversation memory, since Strands itself resets each node's messages/state on a fresh-start restore. See the Strands guide's Multi-Agent Checkpointing section for the full scope, including the cumulative-execution-limit caveat for a Graph/Swarm intended to run on a long-lived thread, and If you need cross-turn conversation memory for why this is upstream-owned and what to use instead.

Storage Architecture

Checkpoints are persisted in Temporal workflow state — the same durable event-sourced storage Temporal uses for workflow execution history. They survive host restarts, worker failover, and Temporal server outages, and are replicated according to your Temporal cluster's persistence configuration.

Every checkpoint has a real size ceiling determined by Temporal's own per-payload limit (roughly 2 MiB by default), with a pluggable External Checkpoint Storage backend available for threads that genuinely need to exceed it. See Checkpoint Sizing and Limits for the full limits table, the config fields that raise them, and how to tell when a thread is approaching one.

Recovery Scenarios

The recovery story for AgentVisor has two rules:

1. State always survives. Checkpoints are persisted in Temporal workflow state, which itself survives host restarts, worker failover, and Temporal server outages.
2. Infrastructure and crash failures restart automatically; everything else is still a caller action. A run that fails because of a worker/heartbeat timeout, worker loss, or a guest/agent process killed by a signal (e.g. OOM, SEGV) is retried automatically by AgentVisor itself — bounded by an attempt budget and backoff, on by default — resuming from the latest persisted checkpoint. A run that fails for any other reason (an agent-raised exception, an activity time-budget timeout, cancellation, or an agent-initiated interrupt()) does not restart: the caller must issue a new run, and the LangGraph checkpointer loads the latest persisted checkpoint on startup so execution resumes from the last super-step.

AgentVisor never retries an agent-raised failure — LLM responses and tool calls aren't generally safe to replay, since replaying could double-charge an LLM call, double-issue a side-effecting tool, or produce inconsistent transcripts. Automatic restart is deliberately narrower than "retry on any failure": it only retries failures that happened around the agent (infrastructure) or that killed the agent process outright (a crash), never a failure the agent itself reported. See temporal.run_restart in the configuration reference for the attempt-budget, backoff, and disable knobs.

What happens in each failure mode

Restarts automatically (bounded: default 3 restarts, exponential backoff starting at 2s capped at 30s, 10-minute total wall-clock budget — see temporal.run_restart):

Failure mode	What survives	Caller action
Guest/agent process crash — killed by a signal (e.g. OOM, SEGV)	Saved checkpoints and any pending writes recorded before the crash persist	None, unless the restart budget is exhausted — then issue a new run as in the non-restarting modes below
Worker/heartbeat timeout or worker loss (includes a host runtime restart, once a worker becomes available again)	Same as above	Same as above
Guest sandbox restart (the gRPC connection to the guest is actually severed, not just the agent process)	Same as above	Same as above
Temporal server outage, once recovered (heartbeat timeout fires as soon as the server can process it)	Workflow state is intact in Temporal's event history	Same as above, unless the outage outlasts the 10-minute restart wall-clock budget — then issue a new run

Does not restart — caller must issue a new run:

Failure mode	What survives	Caller action
Agent-raised exception (agent code itself raises/exits with an error)	Saved checkpoints persist	Issue a new run; the checkpointer loads the latest checkpoint and execution resumes from there
Activity time-budget timeout (StartToClose, 30 minutes) — restarting would just re-burn the same exhausted budget	Same as above	Same as above
Cancellation (caller-initiated)	Same as above	N/A — cancellation is already caller-initiated
Agent-initiated interrupt()	Same as above	Submit a new run with the user's response

Residual risk: a crash during a tool call can still cause that call to replay

Automatic restart resumes from the last checkpoint, including any pending writes recorded via put_writes() for tasks that completed earlier in an in-flight superstep (see Checkpoint Data above). This narrows, but does not eliminate, the replay window:

• A crash during a tool call — after the call has taken effect but before its put_writes() write is durably recorded — can still cause that call to execute a second time on restart.
• The same window opens, independent of any crash, when a superstep's pending writes exceed the configured size/count bounds (silently dropped rather than erroring — see Checkpoint Data above).
• It can also open when LangGraph's put()/put_writes() ordering lets a writes batch arrive host-side before its owning checkpoint has saved (mitigated by a short client-side retry, not eliminated).

Mitigation is agent-level idempotency for side-effecting tools. Workloads that cannot tolerate any replay risk should set temporal.run_restart.max_attempts: 0 (AGENTVISOR_TEMPORAL_RUN_RESTART_MAX_ATTEMPTS=0), which fully disables automatic restart — a restartable failure then surfaces as a run error on the first attempt, identical to pre-restart behavior, and the caller re-issues manually as usual.

Continue-As-New (CAN)

For long-running threads where the underlying Temporal workflow history grows large, AgentVisor compacts state and re-issues any in-flight run automatically. This is a second, narrower path to automatic resume alongside the infra/crash restart described above — it resumes by compacting workflow history and starting a new workflow execution, rather than resuming within the same execution.

1. Workflow history grows large enough to trigger compaction
2. The active run is gracefully cancelled
3. Runs and checkpoints are pruned per AGENTVISOR_TEMPORAL_MAX_RUN_HISTORY
   and AGENTVISOR_TEMPORAL_MAX_CHECKPOINT_HISTORY
4. If the resulting payload doesn't fit the CAN budget, state is shed in a
   fixed order (see [Checkpoint Data](#checkpoint-data) above) until it
   fits, or compaction fails loudly if it still doesn't
5. The workflow continues as a new execution with compacted history
6. The interrupted run is automatically re-issued; LangGraph's checkpointer
   loads the latest checkpoint and execution resumes, re-executing any
   sibling tasks from that superstep whose already-completed writes were
   dropped in step 4

CAN is transparent from the caller's perspective in the common case — the run continues to completion. Runs interrupted via the agent-side interrupt() API still require an explicit client-side resume; only the CAN-induced compaction triggers automatic re-issuance. The step 4 failure path is the exception: it ends the thread's execution rather than continuing it, and is meant to be rare and operator-visible rather than silent. See Checkpoint Sizing and Limits for the exact shedding sequence, the budget it's measured against, and which metric to alert on.

Best Practices

Checkpoint Frequently

Use LangGraph's automatic checkpointing (after each super-step) rather than manual checkpointing.

Keep Checkpoint Metadata JSON-Serializable

Checkpoint state (channel_values) uses LangGraph's own serde.dumps_typed()/ loads_typed() and is not restricted to JSON — LangChain message objects (AIMessage, ToolMessage, etc.) round-trip correctly. Checkpoint metadata, however, is sent as a protobuf Struct and must be JSON-serializable:

# Good - metadata is JSON-serializable
metadata = {"count": 42, "items": ["a", "b"]}

# Bad - function not JSON-serializable
metadata = {"handler": my_function}

Handle Large State

Prefer storing references over inlining large data (images, documents) directly in checkpoint state:

# Good - store reference
state = {"document_id": "doc-123", "summary": "..."}

# Bad - store full content
state = {"document_content": large_blob}

References keep checkpoints small and fast to read/write. If your workload genuinely needs to carry large state inline, see Checkpoint Sizing and Limits and External Checkpoint Storage rather than working around the size ceiling by hand.

Test Recovery

Simulate failures in development:

def test_recovery():
    # Run 1: Execute until checkpoint
    result1 = graph.invoke({"messages": [...]})

    # Simulate restart by creating new graph instance
    new_graph = builder.compile(checkpointer=AgentVisorCheckpointer())

    # Run 2: Resume from checkpoint
    result2 = new_graph.invoke(
        {"messages": [...]},
        config={"configurable": {"thread_id": thread_id}}
    )

    assert result2["step_count"] > result1["step_count"]

Configuration

AGENTVISOR_TEMPORAL_MAX_CHECKPOINT_HISTORY and AGENTVISOR_TEMPORAL_MAX_RUN_HISTORY control how much history is retained per thread — see History Limits. For the full set of checkpoint- and Temporal-related configuration, see the configuration reference.

Debugging

View Checkpoint History

curl http://localhost:8090/threads/{thread_id}/history | jq

Inspect Checkpoint Content

curl http://localhost:8090/threads/{thread_id}/state | jq

Temporal UI

View workflow state in Temporal UI at http://localhost:8233:

1. Find workflow by thread ID
2. View "State" tab for current checkpoint
3. View "History" for all events