08-tooling

Brigade PR #6 — fresh-eyes review + Fable-5 capability check

2026-07-01·review·status: delivered
skill-agent-brigaderay-pluginspr-reviewexecution-evalfable-5

Ask (founder, 2026-07-01 ~16:43 ET, via iMessage, first task on the Fable 5 backend): review the agent-brigade framing as applied in PR #6 on RayDataCo/ray-plugins; does it hold at increased capability level; ready to merge and start developing other brigades?

Method: two zero-context fresh-eyes subagents (per [[feedback_fresh_eyes_subagent_for_own_artifacts]]) — one mechanical merge-readiness pass, one skeptical staff-engineer design critique — plus Ray's synthesis on the capability question grounded in the branch's own measured tier-sweep data.

1. Merge-readiness (mechanical pass)

Initial verdict NOT-READY on two blockers, both trivial. Fixed same hour, commit f2fb84f pushed to the branch:

Passed clean: vocab rename complete everywhere else; exit set enumerated identically across all 11 sites; every doc link resolves; node --check green on all 3 workflows; JSON manifests valid.

Remaining follow-ups (documented, not merge-gating): ticket shape stated 3 ways across the v2 specs (identity / inline payload / external bundle_ref) — settle before any rail/context-prep code; v1 4-field ticket in expo SKILL.md vs v2 bundle-ref (living-doc layering, could use one transition sentence); path-coupled reference workflows (already an owned README follow-up).

Post-fix verdict: READY. Merges clean against main (main only moved because PR #5 / snowflake-cli landed).

2. Design critique (fresh-eyes verdict: sound-with-gaps)

Steelman first (the reviewer's, unprompted): "meaningfully better than the median multi-agent-factory design" — blind test-author as enforced independence; deterministic lint as a hard gate LLM judges can't outvote; execution-eval asking lift-not-correctness and empirically discovering the aggregate-dilution failure; docs that preserve failures instead of laundering them.

The gaps, in order of importance:

  1. "Critic advises, expo decides" is still ~half slogan. The expo SKILL.md claims an information advantage but states no criteria for how it changes a decision; the refire-vs-reroute call (the one decision that genuinely needs cross-station context) has zero written signals. The reference workflow is more damning: the as-run "expo" is verdicts.filter(FAIL && confidence >= 0.6) inside while (round < 2) — no reroute-to-spec path, no kill path, no phase-0 gate; a critic FAIL at 0.59 silently passes. The 6/28 editor's pass flagged exactly this ("make the expo real"); the fire-through demonstrated it once at runtime, but the reference implementation still doesn't embody it. Also structural: execution-eval is never sequenced into the expo's own loop spec.
  2. Ticket/rail contradictions. Two ticket definitions (BUNDLE-SPEC {bundle_ref} vs expo's 4-field record); "no child copies" vs "records exact resolved content" is unresolvable for url/qmd/mcp sources without a content-addressed snapshot store beside the rail; "same ticket → same skill" is falsified by the design's own non-determinism note — rename the guarantee to auditability; snapshots pin the context but not the build environment (station-skill versions, model ids, config). Plus: pull() has no lease/ack → racy under the parallelism the README promises; no compaction policy → tickets become context bombs by construction.
  3. The value gate isn't model-lifecycle-aware. (= the Fable-5 question; see §3.)
  4. 60-skill fan-out breaks, in order: racy rail → human-as-bottleneck at the pass (kill-confirm, max_rounds escalation, phase-0 asks all route to one human with no batching protocol) → context-prep doesn't exist (60 curated bundles is the most labor-intensive stage and bundle quality is the single point of failure — critic verifies against the excerpt, so garbage bundle → confidently-passed garbage skill) → no portfolio-level gate (triggering-precision is judged per-skill; description-space collisions are invisible at n=2, dominant at n=60) → generative/advisory eval is a one-word hand-wave, and that's where Marketing/Sales/Legal — most of the grid — land.
  5. Statistical hygiene: N=3/arm means one flipped sample (±0.33) swamps the 0.15 win threshold; no token/time cost axis on the gate.

3. Does the framing hold at Fable-5 capability? (Ray's synthesis)

Yes — and the branch's own data predicted what stronger models do. The verified tier sweep found two lift signatures: judgment lift erodes as the base model strengthens (variance-analysis fixture D: Haiku +22pp, Sonnet +33pp, Opus 0) and convention lift stays flat at every tier (generate-tests S3: +50pp incl. Opus — house rules can't be inferred). Fable 5 is just the next point on that curve. Three implications, first two now urgent rather than optional:

  1. Gate on the deployment tier + add a retire lifecycle. A judgment-shaped skill that earned advance on Sonnet may be dead weight on Fable — and nothing re-evals on model upgrade. Re-run execution-eval when the base model changes; classify expected lift signature (judgment vs convention) at spec time so erosion is predicted.
  2. Reframe the eval output as "minimum viable tier." The lift matrix already computes it — surface "with this skill, Haiku hits 100%; without it you need Opus" as a first-class result. As top tiers strengthen, the durable value story for most skills becomes conventions the model can't guess + tier derogation (cost arbitrage), not competence the frontier model lacks. This makes the brigade more valuable at Fable-5 level, not less — it's the instrument that tells you which skills survive.
  3. Operational, this week: this session's backend is now Fable 5, and Workflow/Agent station fan-outs inherit the session model — a brigade run from here would burn the founder's included-window cap (50% of weekly limits through Jul 7). Pin station agents to explicit cheaper models for any build runs (the workflows already parameterize model); Fable makes sense only at the expo/judgment seats, if anywhere. Crons stay on manifest models per the standing commitment.

4. Recommendation

Grounding