RDCO multi-agent pipeline = RLHF-shaped (the academic lineage)

Why this is in the vault

Founder spotted on 2026-05-12 20:39 ET that the multi-agent skill build-out pipeline he just greenlit is structurally the RLHF loop. Naming this lineage gives us a reference vocabulary for every future pipeline-design choice. When we have to make a call later (threshold for "critic graduates to autonomous," shape of label aggregation, how to detect critic drift), we can ask "what does the RLHF literature say" instead of inventing from scratch. That's the load-bearing reason to file the concept doc rather than leaving the framing in iMessage history.

The mapping (structural, not mechanistic)

RDCO multi-agent pipeline mapped to RLHF topology:

Loop topology: identical. Founder critique → reward model update → policy iteration → critique → ... until convergence.

The mechanism difference

Textbook RLHF (the InstructGPT / Llama-2 paper / Anthropic Constitutional AI initial-pass shape) trains a parametric reward model via gradient descent on (prompt, chosen, rejected) preference triples. That reward model is then used to optimize a policy via PPO or DPO. The whole thing is differentiable end-to-end.

RDCO pipeline does NOT do gradient descent. The critic is an LLM agent, not a trained neural net. Founder feedback is codified into:

• Critic prompts — the natural-language instructions in pipeline-critic's SKILL.md and per-axis fragments
• Reference exemplars — the canonical-set folder of known-good artifacts the critic compares against
• Structural YAML axes — the deterministic property checks (frontmatter completeness, cross-link richness)

So the reward signal is encoded in PROMPT SPACE, not weight space. Same loop, different machinery.

Closest academic lineage

The technique is closer to:

• Constitutional AI (Anthropic, 2022 paper "Constitutional AI: Harmlessness from AI Feedback") — uses an LLM critic that critiques outputs against a written constitution, then revises. RDCO's per-axis YAML fragments are structurally identical to a constitution's principles. The difference: Anthropic's CAI uses AI feedback to scale beyond human bottleneck; RDCO uses human feedback during bootstrap and graduates to AI-feedback-with-human-spot-checks once the critic calibrates.
• RLAIF (Reinforcement Learning from AI Feedback, Bai et al. 2022) — uses an LLM critic to label preferences for reward-model training. RDCO's "founder-as-critic-bootstrap → critic-codified → critic-runs-autonomously" sequence is the same shift, but happens without the reward-model gradient-descent step.
• Iterative refinement / Self-refine (Madaan et al. 2023) — uses an LLM critic to score outputs and prompt the policy to revise. RDCO's per-run convergence loop is exactly this pattern, multi-agent-split.
• DPO (Direct Preference Optimization, Rafailov et al. 2023) — bypasses the explicit reward-model training step. The RDCO equivalent: skip the "reward model" abstraction and treat the critic's natural-language verdict as direct policy guidance.

The most accurate single label: Constitutional-AI-shaped multi-agent pipeline with human bootstrap labeling. Less catchy than "RLHF" but more precise.

What this implies for our build

Because the topology is RLHF-shaped, several design choices have known-good answers from the literature:

1. Label capture format. RLHF literature converges on (prompt, output, rating, optional-rationale) tuples. RDCO equivalent: per-run scratch dir captures (spec, generated-artifact, critic-verdict, founder-override-rating-if-any, founder-rationale). The structured format makes the /improve consolidation step mechanical instead of judgment-driven.
2. Critic-graduation threshold. When does the founder-as-critic phase end? RLHF literature suggests ~1k-10k preference labels for a trained reward model, but RDCO's per-axis prompt-space encoding works with far fewer examples (Anthropic CAI bootstrapped on ~16 principles). For RDCO, suggest a per-axis threshold of ~10 founder-override labels before that axis graduates to autonomous-critic status. TBD as part of schema work.
3. Drift detection. RLHF systems suffer reward hacking — the policy learns to exploit the reward model's blind spots. RDCO equivalent: the critic can develop blind spots that the policy learns to satisfy without actually meeting ground truth. Mitigation: periodic founder spot-checks of artifacts the critic marked PASS (the auditing layer Tan's ratchet wants). Set spot-check rate at ~5-10% of autonomous runs.
4. Label disagreement handling. When two human labelers disagree, RLHF tools usually average or vote. RDCO has only one labeler (founder), so this is moot — but if a future co-founder or contractor enters the loop, the disagreement handling needs to be designed in. Defer until that surfaces.
5. Pipeline-critic prompt evolution. RLHF literature treats the reward model as a continuously-updated artifact. RDCO equivalent: pipeline-critic's SKILL.md gets continuously updated by /improve from accumulated labels. The skill file IS the reward model state. Versioning + changelog discipline becomes load-bearing.

What this does NOT mean

A few things RDCO's pipeline is NOT, despite the RLHF-shape similarity:

• NOT a fine-tuned model. No weights are being updated. The "model" is the prompt + reference exemplars + YAML axes. Swap-out is a git commit, not a training run.
• NOT reinforcement learning in the optimization-theory sense. No expected-reward maximization, no policy gradient, no value function. The "RL" in RLHF is a misnomer for RDCO; what we're doing is closer to supervised learning from human critique, applied iteratively.
• NOT a substitute for the founder's judgment on what to BUILD. RLHF refines HOW to build; the founder still defines WHAT counts as a good artifact. The constitution / per-axis YAML fragments are the founder's expressed taste.

Mapping against Ray Data Co

• This pattern applies to every skill build-out from 2026-05-12 forward. The 4-seat pipeline is the standard build mechanism for new skills; the canonical-critic-codification-via-RLHF-loop is the standard improvement mechanism.
• Same loop shape extends beyond skills: [[../concepts/2026-04-19-mac-the-monster-anti-cheat-framework-for-data]] (MAC) is already an RLHF-shaped loop where the client (not the founder) is the labeler — client signs off on what counts as production-ready data, those sign-offs codify into the MAC matrix, the matrix becomes the autonomous critic. Filed today: the carve-out-with-counterparty-signature 5th MAC artifact is structurally a preference label.
• [[../concepts/2026-05-11-hq-as-decision-surface-notion-as-data-store]] (HQ click-back rail) is the RLHF labeling surface for non-skill-build decisions. Every APPROVE / ARCHIVE / SPLIT / DEFER click is a preference label, with the rationale text serving as the label's training signal.
• Sanity Check is RLHF-shaped on the content side. Past file-quality articles are the reference exemplars; founder corrections (no-em-dashes, no-derivative, sharp-verdict, X-voice-match memories) ARE the codified critic prompts; ongoing founder verdicts on shared content ARE the labels that update the critic.

Why this matters now vs later

Naming this lineage now (during the architecture-doc phase, before any seats are built) lets us:

• Make schema decisions with academic backing instead of intuition
• Anticipate failure modes (reward hacking, critic drift, label-poverty stalls) before they bite
• Reuse vocabulary across pipeline conversations — "critic graduation," "label volume," "axis-fragment update" become precise terms
• Connect to a growing literature (CAI v2, RLAIF refinements, Self-Refine extensions) for future iteration ideas

Related

• [[../2026-05-12-zach-lloyd-warp-verify-then-build-test-harness-agentic-coding]] — the worked example that founded the pipeline decision (verify-then-build IS the RLHF loop with cv-comparison critics)
• [[../2026-05-12-jaynitx-pattern-recognition-skill-build]] — the mechanism (volume + tight feedback + articulation) is what RLHF formalizes
• [[../2026-05-12-garry-tan-ai-agent-complexity-ratchet-90-test-coverage]] — the temporal commitment (tests + docs + evals every turn) is what makes the label dataset accumulate
• [[../../01-projects/skill-pipelines/2026-05-12-multi-agent-pipeline-architecture]] — the architecture doc this concept formalizes
• [[2026-05-11-hq-as-decision-surface-notion-as-data-store]] — the click-back rail IS the labeling surface
• [[2026-04-19-mac-the-monster-anti-cheat-framework-for-data]] — MAC is an RLHF-shaped loop with the client as labeler (filed earlier)