Multi-Agent Resumability and Checkpointing: What Mature Frameworks Do, and What CAF Should Steal

The question

How do production multi-agent frameworks (LangGraph, CrewAI, AutoGen, Anthropic's multi-agent research system) implement resumable state / checkpointing, and which patterns port to a markdown-skill plugin like CAF?

What we already know (from the vault)

[[2026-06-09-caf-restructure-proposal]] — Move 2 designs engagements/<client>/manifest.yaml as the on-disk state owner: per-phase status, artifact paths, HITL approval records, contract validation scores, and Meta-Council halt state. Synthesizers write contracts and advance the manifest; a resumed session reads it to skip completed phases. The coordinator-holds-state pattern is explicit.
[[2026-06-09-mrbeast-superbowl-puzzle-multi-agent-case-study]] — Founder's lived instance of the coordinator/blackboard pattern: five specialist agents all write into a central coordinator that holds global state and assigns who-acts-next. The beast-bot acceptance signal made failure honest. Independently re-discovered the 1970s blackboard architecture.
[[2026-06-09-commissioning-as-agent-spec-craft]] — The commission checklist item 7 ("who holds shared state?") matches the LangGraph thread_id primitive exactly: specialists write to the coordinator, not to each other. Also surfaces that the Anthropic research system grades the end-state, not the path — which is the CAF acceptance-contract design.
[[2026-04-10-akshay-pachaar-agent-harness-anatomy]] — Identifies state management as harness component 7: LangGraph uses typed dicts + super-step checkpointing; Claude Code's approach is "git commits as checkpoints and progress files as structured scratchpads." The Ralph Loop pattern (Initializer + Coding agent reading progress files each session) is architecturally identical to the CAF manifest bootstrap.
[[2026-05-08-dan-farrelly-background-agents-orchestration]] — The five stable orchestration primitives: durable steps, persistent external state, parallel work coordination, event-driven control flow (HITL pause), and structured observability. Names the key gap: checkpoints save state but leave failure detection, recovery, and duplicate prevention to the developer.

What the web says

LangGraph: thread-scoped checkpoints at every super-step boundary. A StateSnapshot is persisted (SQLite, Postgres, or custom backend) after every node execution, keyed to a thread_id. Resume loads the last snapshot and hands control back to the interrupted node — but critically, the node itself re-executes (not just the next instruction), which mandates idempotency for any side-effectful node. LangGraph Persistence Docs; Durable Execution in LangGraph — Vadim's blog
LangGraph HITL uses the same interrupt() primitive as scheduled pauses. A node awaiting human approval calls interrupt(); a Command(resume=True) restarts it. The checkpoint holds the draft/pending state across process crashes and deploys, making multi-day human pauses safe. Vadim's blog
LangGraph checkpointing has a durability mode tradeoff: sync (safe, slower) vs async (fast, can lose state on crash) vs exit (only on completion, no mid-run recovery). Even sync mode demands deterministic nodes — branching on now() or live randomness breaks resumability. Vadim's blog
CrewAI has no built-in checkpointing. Its @persist() decorator automates state at developer-chosen points but leaves restart logic entirely to the developer. If a four-agent crew fails on agent three, you restart the whole crew or build your own recovery layer. Diagrid Blog; ZenML Blog
Anthropic's multi-agent research system uses external storage + lightweight references, not full state replay. Subagents store work in external systems (filesystem, memory), then pass back lightweight references to the coordinator. When context approaches 200K tokens, agents save their research plan to memory before truncation. Checkpointing + retry logic allows resume from where an error occurred rather than restarting. The coordinator waits synchronously for each subagent batch before proceeding. Anthropic Engineering — How we built our multi-agent research system
"Checkpoints are not durable execution" — the structural gap. Checkpointing says "I saved your state; you handle recovery." Durable execution (e.g., Dapr Workflows) says "your workflow will complete, period." The gap: no automatic failure detection, no automatic resumption, no duplicate prevention, and single-process execution with no distributed task coordination. This is an architectural difference, not a maturity gap — adding a better checkpointer doesn't close it. Diagrid Blog
The Ralph Loop pattern (Anthropic) is the closest production analog to the CAF manifest bootstrap. An Initializer Agent creates a progress file, feature list, and initial git commit; every subsequent Coding Agent reads git logs and the progress file to orient itself, picks the next incomplete item, works, commits, and writes a summary. The filesystem provides continuity across context windows — no special framework infrastructure required. [Akshay Pachaar — Anatomy of an Agent Harness (vault: [[2026-04-10-akshay-pachaar-agent-harness-anatomy]])]

Convergences and contradictions

Convergences — CAF's manifest design maps cleanly onto established patterns:

The manifest.yaml with per-phase status fields is structurally identical to LangGraph's thread_id-keyed checkpoint store, except it uses a named YAML file instead of a database. The thread_id → client-slug/manifest.yaml mapping is 1:1. Both key resumability on reading a single authoritative record to learn where execution stopped.
CAF's synthesizer-writes-contract-then-updates-manifest pattern is the Ralph Loop's Coding Agent → progress file → next agent pattern, applied at the phase boundary instead of the feature boundary.
CAF's HITL gate halts (orchestrator stops, records approval in hitl/ + manifest.yaml, requires explicit resume) match LangGraph's interrupt() / Command(resume=True) pattern exactly — and the Anthropic research system's synchronous coordinator waiting for subagent batches before proceeding.
Subagents store artifacts and return lightweight references to the coordinator (Anthropic's pattern) is what CAF's contracts/ + assets/ dirs with paths recorded in manifest.yaml implements. Full output never clogs the coordinator context.

Contradictions / divergences — where CAF differs from mature frameworks, and why:

CAF does not enforce idempotency at the node level. LangGraph's re-execution-on-resume requires node idempotency. CAF's phase skills, if re-run, could produce different LLM outputs. The manifest design mitigates this at the phase boundary (skip phases marked complete) but not within a phase if a skill partially executes. This is an acceptable tradeoff for CAF's use case (consultant-driven, low concurrency) but would be a liability at scale.
CAF's manifest.yaml is a developer-managed checkpoint, not a framework-managed one. This is closer to CrewAI's @persist() than LangGraph's automatic super-step checkpointing. The Diagrid critique applies: failure detection and recovery are the orchestrator's responsibility, not the infrastructure's. For CAF's single-operator, sequential use case this is fine. For /caf:run-batch at scale it becomes a gap.
CAF has no checkpoint durability modes. LangGraph offers sync/async/exit tradeoffs. CAF writes YAML synchronously via the synthesizer, which is effectively sync mode at phase granularity — a reasonable default for a consulting tool where each phase is a natural breakpoint.
AutoGen's conversational multi-agent pattern has no direct CAF analog — and that's correct. CAF's phases are contract-gated, not conversational. AutoGen is the wrong model for CAF; LangGraph's graph-with-checkpoints is the right mental model even though CAF doesn't use the framework.

Synthesis for RDCO

Adopt: the manifest-as-checkpoint store is validated. The engagements/<client>/manifest.yaml design converges with both LangGraph's thread-scoped state and Anthropic's external-storage + lightweight-reference pattern. The key insight from the frameworks: the checkpoint's job is to be the single authoritative record of "what's done and where to resume." CAF's manifest does exactly this. The phase-level granularity (coarser than LangGraph's super-step) is appropriate — phase boundaries are natural, semantically meaningful breakpoints for a consulting workflow, not an implementation shortcut.

Adopt: the lightweight-reference pattern for artifact handling. Anthropic's research system is explicit: subagents store work externally, return references to the coordinator. CAF's contracts/contract-N.yaml + assets/*.yaml with paths recorded in manifest.yaml is this pattern. The key discipline: the synthesizer writes the artifact to disk before updating manifest.yaml. If it crashes between those two writes, the next run can detect the orphaned artifact by checking artifact path existence vs manifest status. This is worth adding as an explicit invariant in the orchestrator: "a phase is complete only when its manifest entry AND its artifact file both exist."

Adapt: add explicit failure detection at the orchestrator level. The Diagrid critique is honest: checkpointing leaves recovery to the developer. /caf:run-engagement needs to handle the case where a phase's artifact exists but manifest.yaml was not updated (partial write). The fix is a startup integrity check: compare artifact-file existence against phase status and repair the manifest if they're out of sync. This is a two-line check the orchestrator can run at startup, and it closes the gap Diagrid identifies without requiring durable-execution infrastructure.

Reject: true durable execution (Dapr / Inngest primitives) for the current CAF use case. The Diagrid blog is right that LangGraph-style checkpointing is architecturally different from durable execution — but that gap matters when you have distributed workers, multi-tenant concurrency, and process crashes to survive. CAF runs as a single Claude Code session with one consultant. The manifest on disk IS the durable state; process crashes are recoverable by re-invoking /caf:run-engagement with the same client slug. The /caf:run-batch parallel case is the one where this assumption softens — per-client manifests ensure no cross-contamination, but true parallel sub-agent runs could race on writes. Mitigation: each client's manifest is isolated to its own directory, so the race condition doesn't exist across clients; within a client, sequential phases prevent intra-client races. Full durable-execution infrastructure would be over-engineering for v1.

Open follow-ups

Idempotency within a phase: if a leaf skill partially executes before a crash, the orchestrator will re-run the whole phase (since the manifest records phase-level status). Define whether intra-phase idempotency is required or if re-running a phase is acceptable. LangGraph's answer (re-execute the interrupted node, require idempotency) may be too fine-grained for CAF; phase-level re-execution may be the right tradeoff.
Startup integrity check spec: write the concrete invariant — phase.status == complete iff artifacts.<phase-contract-path> exists on disk — and add it as the first action in /caf:run-engagement before any phase logic runs.
/caf:run-batch write isolation: confirm that parallel sub-agents for different clients write only to their own engagements/<client-slug>/ directories and never read each other's manifests. The parallel-sub-agent isolation mechanism (separate worktrees vs separate processes) determines whether file-level isolation is sufficient or a locking scheme is needed.
HITL resume signal design: LangGraph uses Command(resume=True) with a value payload. CAF's async HITL model (no blocking modal per the no-blocking-modal-in-monitoring hard rule) needs an equivalent — an iMessage/channel approval that writes an approval record into hitl/<gate-id>.yaml and sets the manifest hitl_approvals entry. The orchestrator polls manifest state on its next invocation rather than blocking on a signal. Define the approval record schema now to avoid ad-hoc design later.
Cross-client roll-up format for /caf:run-batch: the batch synthesizer needs to read N client manifests and emit a status table. Define the roll-up schema (at minimum: client slug, current_phase, open blockers, meta_council.active_halt) as part of the orchestrator spec before building.

[[2026-06-09-caf-restructure-proposal]] — the full CAF design this brief validates and extends, including the manifest schema, synthesizer responsibilities, and HITL gate halts
[[2026-06-09-commissioning-as-agent-spec-craft]] — the commissioning checklist item 7 (coordinator holds shared state; specialists write to it) and the acceptance-test-before-work discipline
[[2026-06-09-mrbeast-superbowl-puzzle-multi-agent-case-study]] — lived founder instance of the blackboard/coordinator pattern
[[2026-04-10-akshay-pachaar-agent-harness-anatomy]] — the Ralph Loop (progress-file bootstrap), LangGraph state management, and the "git commits as checkpoints" design pattern
[[2026-05-08-dan-farrelly-background-agents-orchestration]] — the five stable orchestration primitives and the explicit argument that checkpoint ≠ durable execution

Sources

Vault:

rdco-vault/01-projects/phdata/2026-06-09-caf-restructure-proposal.md
rdco-vault/06-reference/concepts/2026-06-09-commissioning-as-agent-spec-craft.md
rdco-vault/06-reference/2026-06-09-mrbeast-superbowl-puzzle-multi-agent-case-study.md
rdco-vault/06-reference/2026-04-10-akshay-pachaar-agent-harness-anatomy.md
rdco-vault/06-reference/2026-05-08-dan-farrelly-background-agents-orchestration.md

Web:

multi agent resumability checkpointing caf