01-projects/phdata

caf restructure proposal

2026-06-09·project·status: draft-for-founder-review
phdatacafrestructureproposalproprietarypluginorchestration

The Catalyst Assessment Framework (CAF) is today a library of 106 heavily-prescriptive, plan-only Read/Grep skills run by hand in phase order to emit a build manifest — with no plugin install, no on-disk memory, no live orchestrator, and a v3→v4 migration that left corrupted bodies and stale root docs behind. This proposal makes four moves: (1) package CAF as a private Claude Code plugin distributed on phData's Bitbucket, replacing the bespoke install.sh; (2) add an on-disk engagement workspace — engagements/<client>/ with a resumable manifest.yaml — so contracts persist as artifacts instead of in-conversation YAML; (3) introduce two orchestrator commands (/run-engagement for one client end-to-end, /run-batch over a roster) that carry the Task/Agent/Skill tools the leaf skills lack and hold state in the manifest; and (4) collapse the 106 micro-skills toward ~15-20 objective-oriented skills, each carrying an objective plus a DONE acceptance test plus its load-bearing constraints, letting the model choose the steps. All four are phData-internal; CAF is proprietary and is never published.

1. Current-state diagnosis

The load-bearing facts we are fixing, stated precisely. Where a detail isn't in the founder's UNDERSTANDING.md and needs the live repo to confirm, it is flagged [verify-against-repo].

What CAF is. A library of 106 agent "skills" (markdown prompt-specs) that, run in sequence, walk a consultant through assessing a client engagement and emit a build manifest — a plan of what agents/skills should be built. CAF plans; it does not build. Every skill is allowed-tools: Read Grep; none carry Skill/Agent/Task.

The 7 phases (v4 numbering) + Meta-Council. 1 Engagement Framing · 2 Pre-Engagement Discovery & Readiness · 3 Discovery/Classification/Archetype Tagging · 4 Prioritization/Governance/Platform-Native Inventory · 5 Skill Contracts/Composition/Validation Archetype · 6 Execution & Iteration · 7 Scale/Replicate/Productize. Plus a Meta-Council (m-1 portfolio strategist / HALT authority, m-2 prioritization integrity, m-3 expectation steward) with cross-phase VETO/HALT, and per-phase meta-personas (2-m, 3-m).

The 7 handoff contracts (Contract 0–6) between phases are structural-semantic: they specify the categories of info that must flow (not rigid schemas), carry confidence/provenance metadata and minimum completeness thresholds, and require archetype tags, platform target, and domain rules to propagate unchanged downstream.

The four structural gaps the founder named:

  1. No marketplace plugin. Install is a bespoke bin/install.sh copying skills/ into ~/.claude/skills/ (plus 6 other targets). Root SKILL.md is a catalog doc, not an invokable command; there is no master /catalyst entry point.
  2. No persistence / not resumable. Contracts flow as YAML blobs in the conversation only — no manifest file, no engagements/ dir, no run ledger. A fresh session can't resume unless you paste prior YAML back in.
  3. No live orchestrator. No skill can call another. "Orchestration" = the operator runs 106 skills by hand in phase order. Two synthesizer skills (2-10-orchestrator, 3-10-readiness-synthesizer) aggregate a phase's outputs into the handoff but invoke nothing.
  4. Skills are inert re: real systems. Pure Read/Grep + YAML; they describe deterministic checks (SQL re-execution, lineage) but can't perform them. The co-installed phData ds-*/agent-*/sql-* CLI toolkit is not wired into CAF reasoning.

Migration debt (v3→v4). A codemod renumbered phases (+2) and flattened nested phaseX/ dirs into a flat skills/. Frontmatter is correct; bodies still contain old phase numbers (e.g. 1-1-input-shape-classifier says "Phase 3 (3.1)" where it should read Phase 1), and at least one corrupted threshold — a naive numeric pass turned 0.7 → 2.7 on a 0–1 confidence scale, so a Hard Constraint now reads "confidence below 2.7" (always true; the gate is effectively broken). Stale root docs (SKILL.md, README.md, CONTRACTS.md) are v3 and contradict live v4 (skills/, CATALOG.md, docs/PHASE_TAXONOMY.md); CONTRACTS.md is only half-renumbered (headers say Contract 0 = Phase 1→2, but frontmatter/body say Phase 6→7).

Each leaf skill is a multi-hundred-line spec for one step (Overview, Key Questions, Hard Constraint with a PROHIBITED-from list, Produces, a verbatim Role Prompt, Capabilities, a weighted Evaluation Criteria table, Instructions, YAML I/O) — the over-prescription pattern written for older models, and the reason the count is 106 (Move 4).

2. Move 1 — Plugin packaging (Q1)

Directive: convert CAF to a Claude Code plugin distributed on a private (phData Bitbucket) marketplace. This replaces install.sh's 7-target copy and the bespoke .caf-skills.lock versioning with the native plugin/marketplace flow.

Structure. A marketplace repo containing a .claude-plugin/marketplace.json catalog plus one bundled plugin (caf) that ships the skills and the orchestrator commands from Move 3:

caf-marketplace/                         # phData Bitbucket repo
  .claude-plugin/
    marketplace.json                     # catalog: lists the `caf` plugin + version
  plugins/
    caf/
      .claude-plugin/plugin.json         # plugin manifest: name, version, description
      skills/                            # the collapsed objective skills (Move 4)
        phase-1-engagement-framing/SKILL.md
        ...
        meta-council/SKILL.md
      commands/                          # the orchestrators (Move 3)
        run-engagement.md                # /caf:run-engagement
        run-batch.md                     # /caf:run-batch

marketplace.json declares the catalog (its own name + the plugins it offers); plugin.json is the per-plugin manifest carrying name, version, and description, and bundles the skills/commands. Version lives in plugin.json — that is the replacement for .caf-skills.lock.

Install flow, before → after.

Before (today) After (plugin)
Get CAF clone repo, run bin/install.sh (copies into 7 targets) /plugin marketplace add <bitbucket-git-url> then /plugin install caf@caf-marketplace
Entry point none — operator opens skills by hand namespaced commands: /caf:run-engagement, /caf:run-batch
Update re-run install.sh, manage .caf-skills.lock /plugin marketplace update caf-marketplace (+ optional auto-update); version in plugin.json
Uninstall manual delete across 7 targets /plugin uninstall caf@caf-marketplace

Private-Bitbucket distribution. The docs confirm marketplaces can be added from "any git repository URL (GitLab, Bitbucket, self-hosted)" — /plugin marketplace add <https-or-ssh-bitbucket-url>.git, optionally pinned to a branch/tag with #<ref>. This matches the RDCO pattern (develop in RayDataCo/ray-plugins, port to phData internal Bitbucket) and keeps CAF off the public Anthropic marketplace — it is proprietary and stays on phData's private git. Plugins execute arbitrary code with user privileges, so the Bitbucket-only / access-gated distribution is also the security boundary.

Verified Desktop/Code support statement (grounded in docs.claude.com, fetched 2026-06-09):

Plugins and marketplaces are a Claude Code feature. They install into Claude Code via /plugin, work from any git host including Bitbucket and self-hosted, and namespace their commands/skills by plugin name. The "Claude Desktop" application ships a Code tab that is Claude Code (alongside Chat and Cowork tabs) — so the plugin/marketplace install flow works in the desktop app's Code surface. The consumer Claude apps (claude.ai web and the Desktop app's Chat/Cowork tabs) do not consume Claude Code plugins/marketplaces; they support custom Skills only via manual per-user zip upload (Settings → Features), and custom Skills do not sync across surfaces. Net: CAF-as-plugin targets Claude Code (terminal or the Desktop Code tab). Confirm the exact desktop-app plugin behavior against current docs at install time, since the desktop surface is evolving.

Portability: the leaf skills are plain Read/Grep markdown, so the skill content runs anywhere skills load (a manual claude.ai zip upload is a fallback); only the one-command plugin install and the Task/Agent-bearing orchestrators are Claude-Code-specific.

3. Move 2 — On-disk memory & manifest (Q2)

Directive: introduce an on-disk engagement workspace so a fresh session resumes from a file instead of re-pasted in-conversation YAML. This is the build-manifest-on-disk the commissioning brief called for ([[2026-06-09-commissioning-as-agent-spec-craft]]): the manifest is the STATE OWNER the acceptance contract names — the single source of truth a verifier (or a resumed session) reads to judge "what's done."

Workspace layout (one dir per client, created on first run):

engagements/<client-slug>/
  manifest.yaml          # the resume point (schema below)
  contracts/             # contract-0.yaml ... contract-6.yaml — each synthesizer writes its handoff here
  assets/                # written-out deliverables: classified inventory, build manifest,
                         #   domain-rules registry, prioritization matrix, etc.
  hitl/                  # gate approval records + decision log

manifest.yaml schema (concrete):

engagement:
  client: acme-data
  account_lead: "Ben Wilson"
  framing_source: "2026-06-09 kickoff notes / SOW ref"   # where Phase 1 framing came from
  caf_version: "4.1.0"                                    # plugin.json version that ran this
  created: 2026-06-09T14:00:00Z
  last_updated: 2026-06-09T16:20:00Z

current_phase: 4            # where a resumed session picks up
status:                     # per-phase: not_started | in_progress | blocked | complete
  phase-1: complete
  phase-2: complete
  phase-3: complete
  phase-4: in_progress
  phase-5: not_started
  phase-6: not_started
  phase-7: not_started

artifacts:                  # paths into contracts/ and assets/
  contract-0: contracts/contract-0.yaml
  contract-1: contracts/contract-1.yaml
  contract-2: contracts/contract-2.yaml
  contract-3: contracts/contract-3.yaml
  classified-inventory: assets/classified-inventory.yaml
  domain-rules: assets/domain-rules-registry.yaml

contract_validation:        # did each handoff meet its minimum completeness threshold?
  contract-0: { passed: true,  completeness: 0.92, missing: [] }
  contract-1: { passed: true,  completeness: 0.88, missing: [] }
  contract-2: { passed: true,  completeness: 0.81, missing: [] }
  contract-3: { passed: false, completeness: 0.64, missing: [archetype_tags_incomplete] }

open_blockers:
  - phase: 4
    issue: "Platform-native inventory missing Snowflake warehouse sizing inputs"
    owner: account_lead

hitl_approvals:             # one record per gate cleared
  - gate: phase-1-framing-signoff
    approved_by: "Ben Wilson"
    at: 2026-06-09T14:40:00Z
  - gate: phase-3-classification-signoff
    approved_by: "Ben Wilson"
    at: 2026-06-09T15:55:00Z

meta_council:
  active_halt: false
  last_review_phase: 3
  notes: "m-2 flagged prioritization drift in phase 3; resolved before contract-3."

New synthesizer responsibility. Today a phase synthesizer only emits in-context YAML. The new contract: each synthesizer writes its contracts/contract-N.yaml artifact AND updates manifest.yaml (sets that phase's status, records artifacts paths, writes contract_validation, advances current_phase). This is exactly the implementation-notes pattern ([[2026-05-18-implementation-notes-pattern-for-sub-agent-dispatches]]) — a running on-disk record of decisions/state — applied to the engagement rather than to a code build. A fresh session bootstraps by reading manifest.yaml: it learns the current phase, which contracts validated, what's blocked, and which HITL gates are already cleared.

4. Move 3 — The four levels + two orchestrators (Q3)

The four levels of detail:

(a) and (b) largely exist in the current library. (c) and (d) require NEW orchestrator commands that — unlike the 106 leaf skills — carry the Task/Agent/Skill tools. This is the structural fix for Gap 3 (no live orchestrator). It maps directly onto the coordinator-holds-state pattern from the MrBeast swarm ([[2026-06-09-mrbeast-superbowl-puzzle-multi-agent-case-study]]): the orchestrator is the coordinator that holds global state (in manifest.yaml) and assigns who-acts-next; the phase skills are the specialists that write into shared state, not to each other. It is also where RDCO's Workflow-tool thinking (pipeline/parallel fan-out + a synthesis seat) transfers directly.

(c) /caf:run-engagement <client>

(d) /caf:run-batch <client-list>

Both orchestrators are the only CAF components that need elevated tools; keeping Task/Agent/Skill out of the leaf skills and in the orchestrators is the clean separation (leaf skills stay portable and inert; coordination is concentrated where state lives).

Intra-phase concurrency model (added 2026-06-09, founder Q). Phases are sequential (contract-gated, HITL approval at each boundary). But within a phase the default shape is fan-out → synthesize: a phase's leaf skills are independent analyses off the same inbound contract, so they run in parallel, and the phase synthesizer (2-10, 3-10, …) is the barrier that aggregates them into the handoff. In /caf:run-engagement this is literally parallel() over the leaf skills + one synthesis seat per phase. Known sequential exceptions (the orchestrator must respect these, not blindly parallelize):

Phase 4 needs an explicit autonomy-router (founder's catch — confirmed in the material). Phase 4 is not "run all of 4A–4G on everything." Each work item is routed by its autonomy classification (set in Phase 3: AI-Assist / Hybrid / AI-First). CONTRACTS.md states it outright: "Classification drives A/B/C layer routing." So:

On the naming ("4A/4B/4C/4D/4G/S4…"). It is confusing because one label collides three orthogonal axes: the letters A/B/C are autonomy lanes, D/G are cross-cutting layers, and the S-prefix (S4/S5/S6/S7) is a separate skill-architecture track running alongside every phase — plus 4G is shared across phases 4 and 6. The clean fix falls out of the collapse (Move 4): a Phase-4 of router + 3 lane skills + cross-cutting + synth, named by what they are (assist / hybrid / aifirst / quality / inventory), retires the alphabet soup. Caveat: the exact intra-phase dependency edges ("which skill reads which") are inferred from names + the contracts; confirm against the repo [verify-against-repo].

5. Move 4 — Objective-oriented simplification (Q4)

The over-prescription diagnosis. 106 multi-hundred-line micro-skills is the "written for older models, too prescriptive" pattern. Fable's prompting guidance is that over-prescription degrades capable-model output ([[2026-06-09-fable5-workflow-optimization-memo]]): a long verbatim checklist amputates the model's ability to find a better route, and on Fable can even trigger a silent fallback. A leaf skill that hand-walks every step of one classification is steering, not commissioning.

The collapse PRINCIPLE. Replace prescription with objective + acceptance test + load-bearing constraints, model decides the steps. Each skill carries:

  1. an OBJECTIVE (one sentence: the outcome, not the route),
  2. a DONE acceptance criterion (the acceptance-contract idea — a binary, checkable "this phase's contract validated at ≥ threshold with archetype tags + platform target + domain rules present"; [[2026-06-09-acceptance-contract-template]]),
  3. the load-bearing constraints only (the PROHIBITED-from rules and real thresholds that must hold — e.g. the correct 0.7 confidence floor, not the corrupted 2.7),

and lets the model choose how to get there. Mechanically: a phase's many leaf skills collapse into one "Phase N" objective skill with named sub-objectives — UNLESS a sub-skill is genuinely load-bearing-separable.

DO-NOT-COLLAPSE list (these are real, separable governance concerns; keep them as distinct skills/gates):

Rough target. ~7 phase objective skills + ~3 Meta-Council + a handful of cross-cutting gates (validation-archetype router, domain-rule extractor, HITL checkers) + 2 orchestrators ≈ ~15–20 skills, down from 106. This is an estimate and needs the actual repo to finalize — the real count depends on how many leaf skills turn out to be load-bearing-separable [verify-against-repo].

The framing: fewer skills, each with a sharper objective + a DONE acceptance test, orchestrated by (c)/(d), with state in the manifest.

6. Sequencing & open items

Phased plan (recommended order — lower-risk, value-first):

  1. Phase A — Manifest/memory layer first (Move 2). Lowest risk, unblocks everything else. Add engagements/<client>/ + manifest.yaml and give the existing synthesizers the write-artifact-and-update-manifest responsibility. This delivers resumability without touching the 106 skills yet, and gives the orchestrators (Phase C) a state surface to coordinate against.
  2. Phase B — One objective-skill pilot (Move 4, narrow). Collapse one phase (suggest the most self-contained, e.g. Phase 1 Engagement Framing) into a single objective skill with a DONE test, run it against a real or sample engagement, and confirm it does not lose a load-bearing constraint before collapsing the rest. Validate, then generalize.
  3. Phase C — Orchestrators (Move 3). Build /caf:run-engagement against the manifest from Phase A and the piloted objective skill from Phase B, then /caf:run-batch.
  4. Phase D — Plugin packaging (Move 1). Once orchestrators + the collapsed skill set exist, wrap them in the plugin/marketplace and cut over from install.sh.
  5. Phase E — Full 106→~N collapse. Only after the pilot proves the principle.

(Plugin packaging could go first since it's mechanical, but it adds the least value until there's a /caf:run-engagement worth installing — so it follows the orchestrators.)

What needs the actual repo on this machine (cannot be finalized from UNDERSTANDING.md alone):

Migration-debt cleanup (do alongside, not after):

Risks:

7. Related

Addendum 2026-06-11 — CAF spans the sales→delivery seam (founder framing)

Founder context (iMessage 2026-06-11 evening, after phData sales training; full notes: [[2026-06-11-sales-process-dsa-training]]):

What this strengthens in the proposal:

  1. Move #2 (engagement manifests) gains its sharpest justification. The manifest is the connective tissue between CAF's two halves: sales-time artifacts (assessment outputs, scoping, estimates) land in engagements/<client>/ and delivery RESUMES from the same on-disk state. The official sales process ends Stage 5 with "Sales→Delivery Knowledge Transfer" — the manifest turns that from a meeting into a file handoff. The two dangling artifacts in the sales-process doc (high-level solution approach with no declared consumer; the estimate sheet with no declared producer) are exactly what the manifest would catch and carry.
  2. Phase ordering check: the restructure phases should respect the sales-half / delivery-half boundary — sales-side skills (assessment, scoping support) are the highest-frequency, founder-personally-used surface now that he's DSA-tracked; consider piloting the skill collapse (Move 4) on a sales-half skill first, where he can dogfood weekly.
  3. For AI Workforce engagements (SE=DL), handoff cost → zero and the tooling compounds at both seats — the strongest personal-leverage argument for the build. The founder sells with the first half and delivers with the second, same artifacts end to end.

Still blocked on: CAF repo path/clone to this machine (founder).