CAF Grounding-Asset Storage Pattern

Answers: where do CAF's unstructured grounding/supporting assets (competitive intel, meeting notes, write-ups — per-company, eventually thousands of companies) live so the system scales past one engineer's laptop to a multi-collaborator, multi-harness team, AND how do those blobs link back to the structured catalog?

Reconciles with the locked decisions in [[2026-06-15-caf-architecture-decisions-and-meta-council]], [[2026-06-15-caf-catalog-data-model-erd]], [[2026-06-14-caf-restructure-organizing-brief]], and the isolation/retention research in [[2026-06-17-multi-tenant-governance-shared-vector-store]].

Recommendation (decision-first)

Adopt the standard "object store as system-of-record + catalog pointer row + in-perimeter retrieval index" pattern, landed on the stack you already locked. Concretely:

1. Bytes live in a Snowflake internal stage (object store), one flat namespace, not per-client folders. The stage IS the blob system-of-record. This is the Snowflake-native form of S3 — same object-store semantics, but inside the perimeter you already chose, governed by the same RBAC, and reachable by the same MCP data plane. No second cloud to manage.
2. A thin typed grounding_asset row in the catalog is the pointer + metadata. It holds the stage path, source, sensitivity_tier, retention_class, target_company_id, author, timestamps — and optionally an M:N link to the use_case/citation it grounds. The bytes are in the stage; the catalog row is how everything (UI, MCP, tailoring agent) finds and governs them.
3. Retrieval is Cortex Search over the staged text, in-perimeter. Embeddings + retrieval stay inside Snowflake (per [[2026-06-17-multi-tenant-governance-shared-vector-store]]); no separate Pinecone/Weaviate to audit independently. The tailoring agent grounds itself by Cortex-searching the asset corpus, then citing the grounding_asset row.
4. Authoring ergonomics ride Google Drive as the human front door, with a one-way sync into the stage. Engineers keep writing competitive intel and meeting notes in Drive (where collaboration already works); a scheduled/triggered sync lands new/changed files into the stage + upserts the pointer row. Humans never PUT to a stage by hand. Drive is the authoring surface; the stage is the system-of-record; the catalog is the index.

One line: the blobs are Snowflake-staged objects, the catalog carries a typed pointer row, Cortex Search makes them machine-retrievable in-perimeter, and Google Drive stays the human authoring front door via a one-way sync — so engineers never touch object-store plumbing and the machine never reads from a laptop.

The prescribed industry pattern

This is a solved class of problem: unstructured per-entity knowledge assets at scale that must be both human-collaborable AND machine-retrievable. Mature teams converge on a three-layer split, and the failure mode is always trying to make one layer do two jobs:

The load-bearing principle: the object store is the system-of-record; the database stores a pointer, not the payload; the index is derived and disposable. Putting blobs in a relational/VARIANT column kills you on cost, bloat, and migration; putting metadata in the object store kills you on queryability and governance; trusting the index as truth kills you when you need to re-embed. The pattern is to keep all three and let each do one job.

This is exactly the shape the CAF catalog already takes for structured data — typed spine + VARIANT payload + (planned) Cortex Search. We're extending the same discipline to unstructured blobs: the stage is to grounding assets what VARIANT is to the modal prose.

Honest evaluation of the four candidates

Judged against two requirements that genuinely pull apart: (a) human authoring/collaboration ergonomics and (b) machine grounding/retrieval at thousands-of-companies scale. No single candidate wins both — which is why the recommendation combines two of them by role.

The clean read: Drive and the stage are the two keepers, and they pair as front door (Drive) → system-of-record (stage). S3 and Bitbucket are each right for a different CAF problem (external host; code), wrong for this one.

How it lands on CAF's locked architecture

Nothing here contradicts a locked decision; it slots into the seams already drawn.

• Catalog = typed spine + VARIANT. Grounding-asset bytes are too big and too binary for VARIANT — they go in the stage. The catalog gets a new thin entity grounding_asset (pointer + metadata), same spine-vs-payload discipline as use_case: typed columns for what we filter/govern (stage_path, source, sensitivity_tier, retention_class, target_company_id, author, created_at), VARIANT only if a blob needs loose extracted metadata. Citations stay in the use_case VARIANT payload as decided ([[2026-06-15-caf-catalog-data-model-erd]]); a citation that points at a supporting write-up can carry a grounding_asset_id reference — see open question 2 on whether that link is first-class.

• One shared remote MCP server is the single data-access plane. Add asset operations to it: put_grounding_asset (writes bytes to stage + upserts the pointer row — the only sanctioned write path), search_grounding_assets (Cortex Search), get_grounding_asset (fetch by id → presigned/scoped URL). Both harnesses (Code + Cowork) get these for free, same as the catalog tools. Engineers on Code and reps on Cowork hit the identical plane — no laptop filesystem anywhere in the loop.

• Retrieval wants Cortex Search in-perimeter (per [[2026-06-17-multi-tenant-governance-shared-vector-store]]). A Cortex Search service over the staged asset text keeps embeddings inside Snowflake, inherits Horizon lineage, and avoids a separately-audited vector DB. The tailoring agent grounds by searching this corpus and cites the grounding_asset row that surfaced.

• Human authoring is solved without making engineers touch object storage. The Drive→stage sync (a small scheduled job or an MCP-invokable ingest) is the bridge: author in Drive, sync lands it in the stage + upserts the pointer + (re)indexes Cortex Search. The "locally-mounted Drive" idea was the right instinct (keep authoring where collaboration works) wired the wrong way (mount = single-user, no machine plane). The sync keeps the instinct, drops the brittleness.

Tenancy boundary applied to assets

Apply "open internal, hard-walled external" ([[2026-06-15-caf-architecture-decisions-and-meta-council]]) verbatim — do not invent per-client isolation for grounding assets:

• Internal: flat + shared. One stage namespace, one Cortex Search service, no per-client folders or row-access policies. Any phData engineer/rep can retrieve any company's intel, exactly like the flat Drive today. The target_company_id on the pointer row is a dimension for filtering, not an isolation boundary.
• The only hard wall is the external tailored-demo seam — and grounding assets never cross it. Competitive intel and meeting notes are inputs to tailoring; they are not part of the per-client JSON bundle the external site renders. The bundle carries finished, client-safe use-case content only. So the highest-risk surface stays exactly where Decision A put it (the external bundle), and raw grounding blobs are simply never exported.
• Retention is the one thing to add now (per [[2026-06-17-multi-tenant-governance-shared-vector-store]], the axis the locked decision was silent on): tag every grounding_asset row with source + sensitivity_tier + retention_class at ingest. Meeting transcripts may carry sensitive client content; licensed third-party data (D&B/PitchBook) has redistribution/retention limits. Tagging is cheap on the thin spine and unblocks policy-driven retention later without forcing per-client physical isolation. Isolation stays flat; retention rides on tags — the two axes are separable.

Migration note (forward design — flag the constraint)

This is a forward design, not a migration of a running system. The CAF repo is not even on this machine ([[2026-06-14-caf-restructure-organizing-brief]] standing blocker), and the system isn't live. So "migration" = the build sequence, not a cutover. Don't boil the ocean:

1. Today / unblock authoring (zero new infra): keep engineers writing intel + notes in the existing flat Google Drive. It already works for humans and is the eventual front door — no wasted motion.
2. When the catalog schema lands: add the grounding_asset entity (pointer + the three tagging columns) to the Snowflake DDL alongside the use_case spine. Create one internal stage. Add the three MCP asset tools.
3. Wire the one-way Drive→stage sync: a small scheduled job (or MCP ingest tool) that lands new/changed Drive files into the stage and upserts pointer rows. This is the moment the system stops depending on any laptop.
4. Add Cortex Search last: stand up the search service over the staged corpus once there's a meaningful asset volume to ground against. It's derived/rebuildable, so it can come online after the store + pointers exist.

Each step is independently useful and reversible; nothing requires the live app or the external-demo fork to be settled first.

Open questions for the founder

1. Authoring UX: Drive-front-door + sync (recommended) vs author-direct-in-app later? The Drive sync is the lowest-friction unblock now. If the "live app" fork ([[2026-06-15-caf-architecture-decisions-and-meta-council]] Decision D) gives reps an in-UI editor, some authoring may move there over time. Confirm Drive-as-front-door is the v1 authoring surface, or do you want to wait for an in-app editor?
2. Are meeting notes / write-ups first-class catalog citations, or loose attachments? Two shapes: (a) a grounding asset is just an attachment retrievable by the agent, with citations staying free-text in the use_case VARIANT; or (b) a citation can reference a grounding_asset_id, making the supporting doc a first-class, dedup-able, linkable provenance object. (b) is more powerful for provenance but adds a junction; (a) is leaner. Which level of citation rigor do you want?
3. Sync trigger + cadence: scheduled poll of Drive, or event-driven (Drive change notifications / a "publish to catalog" action the author takes)? Affects freshness vs simplicity.
4. Does any grounding asset ever need to leave the internal plane (e.g., a client wants their own intel write-up in a take-home)? Current design says no — raw assets never cross the external wall. Confirm that holds, since it's what keeps the tenancy model simple.
5. Stage vs S3 — is there a phData reason to prefer S3 (existing data-lake conventions, a partner-stack mandate) over a Snowflake internal stage? Default recommendation is the stage (one cloud, in-perimeter, MCP-reachable); flag if house convention pushes to S3.

Cross-references

• [[2026-06-15-caf-architecture-decisions-and-meta-council]] — tenancy, datastore, harness/MCP, demo-site decisions
• [[2026-06-15-caf-catalog-data-model-erd]] — the catalog spine + VARIANT model the pointer row joins into
• [[2026-06-14-caf-restructure-organizing-brief]] — data-plane/agent-plane split, one-MCP-server rationale, repo blocker
• [[2026-06-17-multi-tenant-governance-shared-vector-store]] — in-perimeter Cortex Search, isolation vs retention as separable axes, tagging