CAF ecosystem map + brigade-restructure read

Context: founder passed 4 phData-internal zips + a Miro interface diagram (2026-07-01 ~22:49 ET) and asked: can we restructure CAF into the Agent Brigade structure; jobscraping as a station in a Company Research brigade; the C4→ACF and C4→Catalog handoffs need defining; all assets track back to the cellar. Privacy: code lives in ~/Projects/phdata-private/ only; generated code → private repos only. This doc is the synthesis of 4 sonnet Explore agents' code-level maps.

1. The four codebases, load-bearing facts only

CAF plugin (phdata-ai-wf-plugins) — TWO implementations coexist

The single most important finding: the repo contains two independent, non-interoperable CAF implementations that agree on the 8-phase concept and differ in nearly every mechanism:

Shared invariant semantics (must survive any restructure): platform_target frozen at P1 · archetype_tags frozen after P3 · domain_rules append-only. Three human-mandatory gates survive even ai_first mode (framing, deliverable review, productization). Half-CAF (P1–4, terminal at the Build Manifest) is already a severable first-class truncation point.

Snowflake frontend (phdata-caf-snowflake-frontend)

Next.js 16 on SPCS (a generic phData template, not re-skinned); no app-level auth (pure Snowflake RBAC, and the caller's-rights plumbing exists but no route uses it); no DDL anywhere — the schema exists only as live objects, reverse-engineered from DML. The same dataset is stored three divergent ways (CAF_ASSESSMENT.catalog VARIANT [the one the UI reads] · normalized CAF_USE_CASE rows + a second payload VARIANT · a dead hand-authored MGIC dataset). 4 live engagements: bossci, mbusa, mgic, sobi — no two use the same Contract-3 shape.

The ingestion seam (the founder's named pain) is exactly as bad as suspected: a shell script PUTs an engagement folder file-by-file (failures swallowed) → a stored proc reads 5 magic filenames via a COPY-INTO-temp-table hack → regex/string-split parsing of STATE.md's markdown table → six normalize_* functions that hardcode two shapes literally commented "MBUSA format" / "Sobi format" and silently emit empty data for any third shape → $1.2M–$2.4M LLM prose parsed by regex, fallback (0,0) silently. Zero tests (vitest wired, no test files). The proc is checked in twice with no sync. The in-app file editor can desync stage-from-catalog (saves don't re-trigger ingest) and can't edit .yaml — the one file that matters.

Embarrassment-grade quick wins regardless of restructure: every client's downloadable executive brief is hardcoded "Safe-Guard Catalyst Assessment"; every "Talk to an expert" mails one hardcoded AE (Jordan Reyes) instead of the engagement's account lead; internal phase YAML + raw intake notes are browsable from the sales-facing app via the file-browser route; Industry filter is a ghost (field never populated); ~2,100 lines of dead Mitratech-era fork still shipped.

ACF (asset-creator-framework)

A pure spec, not a system: 104 markdown files, all one filesystem timestamp (single generation pass 2026-04-28), no code, no schemas, explicitly self-described as "the canonical build spec" / WBS for future engineering. Declares it consumes a "CAF Engagement Manifest" (note: NOT called "C4 Build Manifest" — naming mismatch with CAF) with 8 named categories — and no schema for it exists anywhere, despite skills describing precise-sounding validation against it. The 32 "agents" have byte-identical Process sections (verified by hash) — they're really ticket templates (role + skill-list + I/O), not distinct designs; the 70 skills are genuinely atomic station candidates; A-01 Conductor = expo; A-02 Manifest Reader = steward; the 5 mandatory gates = critic stations; 2 of its 4 self-declared vocabulary levels (18 Use Cases, 19 Workflows) are phantom — asserted counts with zero artifacts.

Jobscraping (phdata-jobscraping)

A real, working internal GTM tool — sensitive: active_clients.json names 55 real clients w/ account-lead emails; 15,692 scraped jobs / 41 companies in a local SQLite db. Two layers: ATS auto-detection (15 platforms) + per-ATS scrapers (mostly hitting ATS JSON APIs; Playwright w/ stealth patches for SPAs; a LinkedIn module using a personally-authenticated browser profile = ToS risk to flag). Enrichment pipeline: deterministic tech-stack extraction + clustering + GTM insight cards + a styled HTML report — but the salary/FTE-investment figures are LLM estimates rendered with the same visual authority as scraped data. Fully standalone: no CAF wiring, everything lands locally, nothing reaches a shared store. Live data-quality bug: Comcast and comcast are two companies (identity resolution unowned). Dormant gold: signal_scrapers.py sketches four sibling stations — InvestorRelations (→ the 10-K/10-Q cellar category), GitHub org, USPTO patents, FDA.

2. The brigade fit — it's latent everywhere

The restructure is less an imposition than a naming of what's already half-built, three separate times:

Granularity call (the central design decision — REVISED after founder's runtime data, 22:59 ET): founder runs (A) himself: /caf conductor → one subagent per phase → each phase agent runs its 3–15 skills sequentially, and P3 + P4 additionally loop per use case — total ≈ 6 hours per company (colleagues report ~30 min on the old monolith; 12× slower = adoption-killing even though rigor is higher). The serial-inside-phase design is the bottleneck, so the granularity answer is two-tier:

• Phases stay the contract boundaries (C1–C7 unchanged, gates unchanged) — engagement-level tickets.
• Inside P3/P4, the USE CASE becomes the unit of work — a micro-ticket on an inner rail. The per-UC dimension battery (classify, archetype-tag, risk, value, etc.) runs as a parallel fan-out: wall-clock = slowest use case, not sum-of-use-cases. With a ~10-agent concurrency cap, a 20-UC engagement's P3/P4 loops drop roughly an order of magnitude. The 106 skills stay the stations' internal procedure; nothing about the contracts changes — this is pure execution-shape.

Use-case refinement (founder's second pain, same message): regenerating/adding ONE use case with new context currently requires re-running a full half-CAF. If use cases are tickets, this dissolves rather than needing a bespoke "refinement station": each UC ticket carries its own context pointers + append-only work log, so "refine UC-x with additional context" = steward appends the new context to that ticket → re-enqueue it through the P3/P4 stations → its outputs re-merge into the Build Manifest/catalog (with the invariant rules enforced: archetype re-tagging allowed only per the frozen-after-P3 policy, domain_rules appends fine). One ticket re-run, minutes not hours, full provenance of the refinement in the work log. This is the mutable-ticket-is-the-build-record model earning its keep on a real operational need.

3. The two poorly-defined seams — now measured

• C4 → Catalog: better defined than feared, but not wired. CAF-(B) already specs a typed catalog block (manifest schema v1.2.0: catalog.meta + catalog.ucs[], "read directly by the catalog app with no runtime transformation"). The deployed frontend doesn't consume it — it ingests the 5 magic files with per-client hardcoded normalizers instead. So the fix is not to invent the port; it's to adopt the one CAF-B already defined: typed manifest in, schema-validated (fail loudly, never silently zero), 1:1 map into the DB. That single change deletes the entire normalize_* family, the dollar-regex, the STATE.md scraping, and the engagement-config side-channel — and is unit-testable with fixtures (the seam currently has zero tests). This is literally our TICKET-CONTRACT pattern (deterministic Gate A at the port, enforced both sides) applied to that boundary.
• C4 → ACF: genuinely undefined on both sides. ACF names a manifest CAF doesn't emit under that name and provides no schema; CAF's Build Manifest wasn't designed with ACF's needs (stakeholder map, ROI scenarios, operating model) in mind. Since ACF is a spec, this seam is greenfield: define it as ACF's menu — the brigade-native move. And building ACF itself (70 skills from a WBS) is literally the skill-agent-brigade's use case — the factory builds the ACF brigade from its own spec.

4. Recommended shape + sequence (Ray's read, for founder reaction)

Canonical-implementation call: (B) as the base — newest, manifest-model ≈ our ticket model, Snowflake VARIANT backend already spec'd, steward already built in. Keep from (A): gatekeeper + meta-council as distinct expo tiers (add reroute-to-spec + kill to the gatekeeper's exits, mapping escalate→human pause exactly like our brigade), the 3 autonomy modes, and the 106 per-skill evals (ready-made execution-eval fixtures). Reconcile the 104-vs-106 count while at it.

Sequence:

1. C4→Catalog port first (production-adjacent pain, 4 live engagements): typed Build-Manifest/catalog schema + rewritten ingest that validates-and-fails-loudly + fixture tests. Small, high-value, self-contained. (Bundle the quick wins: Safe-Guard branding, AE hardcoding, file-browser exposure.)
2. Company Research brigade: merge jobscraping + its 4 dormant signal-scraper siblings + CAF-(B)'s Step-0 research routines (the same roster conceived twice!) into one brigade whose steward owns canonical company identity (fixes the Comcast/comcast class) and whose outputs land in the cellar — not local dirs. Cellar v1 = the engagement context tree; the Snowflake VARIANT backend gives cellar + rail a shared enterprise home later.
3. CAF brigade vocabulary pass: phases=stations, gatekeeper=expo (+2 exits), manifest=ticket (+append-only work log), Step-0=steward, contracts=menu entries. Mostly renaming + gap-filling, not rebuilding — CAF-(B) is already brigade-shaped.
4. ACF last: define its menu (the C4→ACF contract) first, then build its 70 stations through the skill-agent-brigade — the factory's first big production run.

5. Questions for founder (asked 2026-07-01 night)

Answered live 22:59 ET: founder created most of this himself (hence the brigade-vocabulary convergence); he personally runs (A) via /caf (~6 hr/company; colleagues ~30 min on the old monolith); the runtime + single-UC-refinement pains are now section-2 drivers.

1. A-vs-B (refined): you run (A); (B) carries the newest design work (manifest, Snowflake backend, Step-0, catalog block). Is (B) your in-progress successor design — i.e., can we declare it the base for the brigade restructure and fold (A)'s gatekeeper/meta-council/evals into it? And does phdata_caf_tools (audit-engagement, codemod-gen-agents, phases.toml) live in another repo we should have?
2. Blast radius: is this restructure a founder-sandbox proposal to bring back to the team (Tiger Team / AAI), or are others consuming these repos live such that we must stay backward-compatible? (4 live engagements in the frontend; licensed IP.)
3. The C4 schema decision: treat CAF-(B)'s catalog block v1.2.0 as the seed of the typed Build Manifest and extend it for ACF's needs — one manifest with two views (catalog view = estimate-tolerant, delivery view = precise + domain-rule-attached, per CAF's own fidelity split)? Or two separate artifacts?
4. Sequence sign-off: agree with 1→4 above, or reorder?

Sensitive flags surfaced: jobscraping DB carries real client + account-lead data (private-repo discipline extra-important); LinkedIn scraping via personal auth = ToS exposure; LLM-estimated salaries presented as data in client-facing-style reports; frontend file-browser exposes internal work product to anyone with catalog access.

Full code-level maps produced by 4 Explore agents this session; raw codebases at ~/Projects/phdata-private/ (local only, no remotes).