06-reference/research

binary run output accumulation storage patterns

2026-10-10·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)·! medium
storage-architecturegit-lfscloudflare-r2content-addressed-storagescribble-works

Binaries that accumulate forever belong in an object store behind a committed hash manifest, and Git Large File Storage is the one option whose bill cannot be pruned

The question

Verbatim as filed on 2026-07-29:

"For a repo that accumulates binary run-outputs from every client engagement forever, what do established patterns (Git LFS, artifact-store-only with pointer files, content-addressed storage) trade off in practice at multi-year scale, and which fits a solo-founder-operated consultancy repo best?"

Context: the question was auto-promoted from open decision D2 in [[2026-07-27-caf-technical-architecture-and-backlog]], which flagged that run outputs accumulate every engagement forever and that "retrofitting is painful," ahead of the CAF-ART-1 ticket.

Note on re-aiming

The original framing is stale in two ways, so this brief answers the technical question but points it at live surfaces. First, "client engagement" and "solo-founder-operated consultancy repo" describe a business that is dormant: the founder's 2026-09-23 ruling treats Ray Data Co as dead unless a side bet earns, and the phData role forbids taking clients or selling services. Second, the "CAF" name was retired 2026-08-10 and that work is now framed as Organizational Intelligence. What survives unchanged is the problem shape: a git repo that accrues binary outputs permanently. Two live surfaces have exactly that shape, so the recommendation is aimed at them: the Organizational Intelligence platform work, whose [[fabric-spec-v0]] specifies a Store holding "landed artifacts" with lifecycle "Append-only, forever," and Scribble Works, which accumulates generated game PDFs and art blobs in Cloudflare R2 and Workers KV.

What we already know (from the vault)

What the web says

All vendor figures below were read from the live page on 2026-10-10 and are dated accordingly.

Convergences and contradictions

Synthesis for RDCO

Recommendation: artifact-store-only, with a committed content-addressed manifest. Concretely, Cloudflare R2 for the bytes and a sha256 plus byte-size manifest in git, which is the pattern Scribble Works already ships. Do not adopt Git LFS. Do not adopt git-annex or DVC either, and this one is a closer call that deserves its reason stated: both genuinely fix the pruning hole, since git annex dropunused and dvc gc remove old blobs without rewriting git history. They lose on single-operator burden and on recoverability. Each adds a tool that must be installed and kept working on every machine and in continuous integration, each stores its cache as hash-named blobs rather than by filename, so recovering a file without the tool means reconstructing directory structure from metadata, and neither tool's documentation covers continuous-integration setup at all. A text manifest plus a bucket needs no install anywhere and is readable with cat. The deciding factor against Git LFS specifically is not cost, though R2 wins there at $0.015 per GB-month against GitHub's listed $0.07 per GiB with free versus $0.0875 per GiB egress. The deciding factor is that R2 lifecycle rules can expire or down-tier a prefix on a schedule and Git LFS objects cannot be pruned at all short of deleting the repository. For a store whose declared lifecycle in [[fabric-spec-v0]] is "Append-only, forever," the ability to change your mind about retention later is the whole ballgame.

The condition under which I would pick differently is narrow and worth naming precisely. If the artifacts must be byte-reproducible from a checkout alone, with no network credentials and no second tool, then Git LFS's integration is worth its quota ceiling. That is the handoff case: someone receives the repository and nothing else, and must get real bytes. If that becomes a requirement and total lifetime volume stays inside the documented 10 GiB storage inclusion with few CI fetches, LFS is defensible. Neither live surface has that shape today. Scribble Works serves PDFs to the public over R2's URL, so the bytes are already a network fetch by design, and the Organizational Intelligence Fabric is specified as a port set precisely so the storage adapter can be swapped.

For Scribble Works, the action is mostly to stop the bleeding and name what already works. The R2-plus-manifest rail for game PDFs is correct and should be stated as the standing pattern rather than left as an emergent property of the launch runbook. Two concrete gaps follow from the verified figures. First, generated art currently lives in the CUSTOMIZE KV namespace because the Pages R2 binding breaks Function publish, and KV stored data is $0.50 per GB-month against R2's $0.015, roughly 33 times more. That is fine for small blobs carrying an expirationTtl and wrong for anything retained. Keep KV for expiring runtime art, and if any generated art becomes durable catalog content, it belongs in R2 behind a Function route that fetches over the public URL or the S3 API rather than a binding. Second, the repo still carries 688 MB of history from the screenshot incident. That is a one-time git filter-repo plus force-push, which the remediation note correctly escalated as a founder decision rather than a cleanup, and it should stay escalated rather than quietly executed.

For the Organizational Intelligence platform work, the recommendation slots in as a Store adapter and needs no spec change. [[fabric-spec-v0]] already partitions Store from Queue and already declines to pick a backend. The adapter contract this brief argues for is: artifact bytes go to an object store, the repo or catalog holds a typed pointer carrying path plus sha256 plus size plus a retention class, and the retrieval index is treated as derived and rebuildable. That is identical to the recommendation in [[2026-06-17-caf-grounding-asset-storage-pattern]], which named a Snowflake internal stage as the object store for in-perimeter reasons. The pattern is backend-agnostic, so both readings hold: R2 where RDCO owns the stack, a Snowflake stage where the perimeter is the governing constraint. The retention class field is the piece that was missing, and it is what makes a lifecycle rule expressible later instead of requiring a migration.

The cheapest reversible first step, and it is cheap enough to do before deciding anything: write the manifest, and gate the commit. Before any storage choice, emit a plain-text manifest next to every run output recording relative path, byte size, and sha256. A manifest is a few kilobytes of text, it is readable with no tooling installed, it costs nothing to carry, and it converts every future migration from a judgment call into a mechanical copy-and-verify. The 2026-09-02 screenshot move is the proof: it succeeded safely because it was "Copied, byte-compared (368/368, cmp clean), then git rm -r --cached," and "Nothing was deleted before the compare passed." That is a manifest discipline applied after the fact. The second half of the step is a pre-commit size gate that refuses any new blob over a chosen threshold, which is the only control that actually prevents the retrofit problem D2 was worried about, because it stops the history from accruing objects in the first place.

Why this is in the vault

This answers open decision D2 from [[2026-07-27-caf-technical-architecture-and-backlog]], which gated the CAF-ART-1 ticket, and it supplies the Store-adapter contract that [[fabric-spec-v0]] deliberately left unspecified. It also settles two live Scribble Works calls with verified numbers: whether durable generated art belongs in Workers KV or R2, and whether the existing R2-plus-manifest rail should be named as the standing pattern.

On D2's status: the decision is not still live, and it is not quite settled on the record either. Its original form is moot, because caf-artifacts was scoped around client-confidential engagement material for a consultancy line that is dormant and a role that forbids clients. Its substance was effectively answered twice before this brief: once by [[2026-06-17-caf-grounding-asset-storage-pattern]], which rejected version control for blobs explicitly, and once in shipped code, by the Scribble Works R2-plus-printables.manifest.json rail with its hash-matching launch gate. What this brief adds is the vendor-verified reason the answer is right rather than merely conventional, namely that LFS objects cannot be pruned and R2 lifecycle rules can. The genuinely open descendant is not D2 but CAF-ART-4-shaped retention policy, and R2 object lifecycles plus a retention-class field on the pointer make that mechanical rather than architectural.

Open follow-ups

Related

Sources

Vault:

Web, all read 2026-10-10:

Explicitly not verified, so not relied on above: the retrofit and history-rewrite guidance for git-annex and DVC specifically (neither tool's migration page was read from a live source today, so the brief's retrofit claims rest on GitHub's documented filter-repo requirement and the Busseneau measurement, not on git-annex or DVC documentation); git-annex special-remote support for S3 or R2; whether GitHub's $0.07 and $0.0875 per-GiB rates are currently charged, given the calculator's "in the future" framing.

Searched and not found, recorded so it is not re-searched: no primary writeup on how the artifact-store-plus-manifest pattern aged over multiple years; no primary writeup on content-addressed-storage operating cost for a solo operator; no "Git LFS considered harmful" post from the Mercurial community appears to exist, so the frequently-cited reference should be treated as misremembered. No paywalled source was used; nothing was blocked or retried.