06-reference/concepts

commissioning as agent spec craft

2026-06-09·reference
commissioningagent-specorchestrationdispatchconcept

Commissioning as agent-spec craft

Writing a great commission and writing a great agent dispatch are the same skill: you define the finish line and a pre-registered acceptance test, you stay method-agnostic everywhere the maker knows the method better than you, and you add staged verification gates in proportion to how catastrophic and irreversible a failure would be. The canon of great commissions — from the 1714 Longitude Act to the 2025 Anthropic multi-agent research system — keeps re-discovering the same handful of marks, because a patron specifying an outcome to a maker and a dispatcher specifying an outcome to a sub-agent are running the identical control problem. This brief extracts those marks, anchors them to a lived multi-agent run (the MrBeast Super Bowl swarm), and maps them onto what our harness already enforces versus where the gaps still are.

This extends the Fable-5 memo's "I no longer steer; I commission" thesis ([[2026-06-09-fable5-workflow-optimization-memo]]): commissioning a studio and signing off on the work is exactly the move, but a commission is only as good as its acceptance test and the independence of its verifier.

2. The spectrum

There is one axis, set by two questions: how precisely can you pre-specify the outcome, and how catastrophic/irreversible is a failure?

PURE OUTCOME-COMMISSION + TRUST  <-------------------------------->  TIGHT SPEC + STAGED ACCEPTANCE GATES
   (prizes, art, broadcast search)                                      (warships, Apollo subsystems)

   Longitude Prize    Orteig    XPRIZE    DARPA    Sistine ceiling    Apollo    Eiffel    Naval commissioning
   InnoCentive        Netflix   Kaggle    bug bounties               (one binary top spec, gated subsystems)

Where agent dispatch falls: most of our work sits left-of-center but instrumented. The outputs (newsletter voice, vault notes, paper-trade theses) have no objective in-transcript oracle, so we commission outcomes and stay method-agnostic — but the blast radius of a bad paper-trade or a fabricated channel reply is high enough that we bolt on a fresh-eyes verifier gate. The position is chosen deliberately: the Sydney Opera House disaster (below) is what happens when you use prize-style trust where ship-style gates were needed.

3. The canon, by lesson

Example (dates) The acceptance criterion The transferable lesson
Longitude Act / Board of Longitude (Act 1714; Harrison's H4 sea trial 1761-62; settled 1773) Tiered, tied to a real transatlantic test voyage: £20,000 for longitude within 0.5 degree (~30 nm), less for 1 degree and 2/3 degree. An actual sea trial measuring real error, not a paper proof. Lock the verifier, not just the criterion. The spec was clean; the judge was conflicted (a Board of astronomers favoring the rival lunar-distance method) and moved goalposts for years. A biased acceptance authority nullifies a perfect spec.
Orteig Prize (offered 22 May 1919; won 21 May 1927) Binary and unambiguous: a continuous non-stop NY-Paris flight. $25,000 purse. Procedural guardrail: 60 days must elapse between the trustees' acceptance of an entrant's papers and takeoff. A single vivid finish line mobilizes disproportionate effort — but a pure-outcome spec with no safety floor rewards escalating risk. ~$400k of effort chased the $25k purse (~16x leverage). Before Lindbergh, six died in three crashes and three more were injured in a fourth.
Ansari XPRIZE (announced 1996, funded 2002; won 4 Oct 2004) Fly to >=100 km with one pilot + ballast for two more adults, then repeat within 14 days replacing <=10% of non-propellant mass. Independently measured altitude. Encode the real business goal into the test, not a proxy. "Reach space" rewards a throwaway rocket; the reusability/14-day constraint forced the operable vehicle the sponsor actually wanted. The constraints ARE the spec.
DARPA Grand Challenge (desert) (2004 zero finishers; 2005 Stanley wins) 2004: ~142-mi Mojave course autonomously in <10 hrs, $1M. 2005: 132-mi course in <10 hrs, $2M. Hard pass/fail, no teleoperation. A zero-finisher round is data, not failure. 2004 had no finisher (best: CMU Sandstorm at 7.32 mi). DARPA held the same hard bar, doubled the purse, re-ran — and 4 vehicles finished within the 10-hr limit in 2005 (a 5th, TerraMax, completed the course but blew the time cap and was prize-ineligible). The time cap was itself a real, non-gameable gate.
DARPA Urban Challenge (3 Nov 2007) Complete ~55-60 mi of three urban missions in <6 hrs while obeying traffic law, scored with a penalty system for violations and dangerous behavior. $2M/$1M/$500k. When the task involves interaction or judgment, pair the binary finish line with a rubric that grades HOW. A pure "finish fastest" metric in a multi-agent environment rewards reckless driving. The penalty layer was essential.
Netflix Prize (2 Oct 2006; won 26 Jul 2009) Beat Cinematch RMSE by exactly 10% -> RMSE <= 0.8572 on a sequestered hidden test subset entrants never saw. Winner hit 0.8567 (10.06%). An exquisitely specified metric can still be the wrong target (Goodhart). Netflix never deployed the winning ensemble — too complex to ship, optimizing a number that drifted from real engagement. The open dataset was also de-anonymized; the sequel was cancelled in 2010 over privacy harm.
Kaggle public/private leaderboard (2010-present) A pre-declared metric scored on a public leaderboard (sample of test data) during the contest, with the winner decided on a private portion scored on held-out data the maker optimized against blind. Verify on ground the maker never optimized against. The held-out split is the structural antidote to metric-gaming; the size of the public/private "shake-up" tells you how much gaming was happening.
Bug bounty programs (modern ~2013-present) A reproducible proof-of-concept inside a declared scope, paid in tiers to validated severity (typically CVSS). Reports lacking repro steps are closed as not-applicable. Coordinated disclosure with a fix window (~90 days). Bound the scope explicitly and make the acceptance test a reproducible artifact the verifier runs independently. The cleanest trust-but-verify loop in the canon. Vague scope = beg-bounty noise and severity disputes.
Sistine Chapel ceiling (contract 8 May 1508; unveiled 31 Oct 1512) Soft/aesthetic — the patron's eye. Julius II climbed the scaffold to inspect; final acceptance was the public unveiling. No measurable spec. When the maker out-experts the patron in the method, specify the outcome envelope and let the maker redesign the content. The masterpiece came from Julius REVOKING his prescriptive 12-Apostles brief. But payment disputes (1510-11) stalled the work — even trust-based commissions need payment cadence tied to delivered work.
Sydney Opera House (competition won 29 Jan 1957; Utzon resigned Feb 1966; opened 1973) Two-stage, and the gap is the lesson: Stage 1 was a subjective judgment of concept potential (Saarinen pulled Utzon's entry from the reject pile); Stage 2 was the brutal objective reality of geometry, cost, and schedule the concept was never tested against. The more you select on a thin/inspiring spec, the more you must instrument the build with downstream feasibility gates. Est. A$7M / 4 yrs; delivered ~A$102M / 14 yrs (~14-15x over, a Guinness record). Trust without staged gates let unbounded method-risk run.
Eiffel Tower (brief 2 May 1886; concession signed 8 Jan 1887; opened 1889) A precise quantified spec (300m, 125m base, iron, ready for the 1889 fair) PLUS named conditions (lifts, lightning). Feasibility WAS a selection criterion — rivals were rejected for lacking buildable detail. When you CAN pre-specify precisely, do — and push method-risk onto whoever controls the method, paid with the upside. The concession (Eiffel finances it, keeps the revenue) capped the patron's liability and made the build police itself. Delivered on spec, on time, no overrun. The instructive contrast IS the absence of a disaster.
Apollo program (goal stated 25 May 1961; Apollo 11 landed 20 Jul 1969) Layered: a binary verifiable top-level criterion, decomposed into subsystem performance specs each gated by qualification tests, design reviews, and uncrewed-then-crewed flight tests. Decompose one unambiguous top-level outcome into independently-specifiable, independently-testable subsystems. A clear binary finish line is what lets thousands of contractors self-organize against it in parallel.
Anthropic multi-agent research system (published 2025) An LLM judge scores each final output against a rubric (factual accuracy, citation accuracy, completeness, source quality, tool efficiency) as a 0.0-1.0 score + pass/fail. Evaluate the FINAL STATE, not the process. Grade the end-state, not the path — capable solvers find alternative valid routes. Process-grading both amputates the distant-solver advantage and rewards going-through-the-motions. Multi-agent uses ~15x the tokens of chat, so reserve it for high-value parallelizable work.

4. The seven marks of a good commission

Each mark, one line, with the example that proves it:

  1. Specify the finish line, not the route. State the outcome; stay method-agnostic where the maker knows the method better. (Lindbergh's solo bet beat the funded consensus under Orteig's method-agnostic spec; Julius's masterpiece came from revoking the prescriptive brief.)
  2. Write the acceptance test before work starts — objective, measurable, verifiable by someone other than the maker. (Longitude's 0.5 degree on a real voyage; Netflix's exact 10% RMSE; a reproducible PoC in scope.)
  3. Pick the metric that tracks the REAL goal, and encode deployability/safety INTO it (Goodhart's Law). (Netflix's RMSE winner was never deployed; XPRIZE's <=10%-mass-replaced and DARPA Urban's traffic-law penalties exist to make the proxy match intent.)
  4. Verify on ground the maker never optimized against; never let the producer be sole judge of its own output. (Kaggle's public/private split; the fresh-eyes-reviewer pattern.)
  5. Make it audacious AND believable, and bound the scope. A finish line no maker believes is reachable attracts no serious effort; an over-broad scope produces noise. (Orteig's first 5-yr term lapsed because the tech wasn't ready; bug bounties work because scope says what's in-bounds.)
  6. Hold the bar steady across iterations and tie payment/authority to verified increments. A zero-finisher round is data; lowering the bar destroys the signal. Make acceptance a distinct, accountable event. (DARPA 2004->2005 held the same bar; ships separate "delivery" from "commissioning" so authority transfers at a defined moment.)
  7. Scale verification to the failure cost — pair the binary finish line with a HOW-rubric and staged gates where a miss is catastrophic. (Apollo's gated subsystems caught nothing until Apollo 1 forced the redesign the gates existed for; Sydney's missing feasibility gate cost ~14x.)

5. Anchor case study: the MrBeast Super Bowl swarm

The founder's own lived instance ([[2026-06-09-mrbeast-superbowl-puzzle-multi-agent-case-study]]), possibly predating "Ray." The commission was one fixed requirement — get the code (solve the MrBeast Super Bowl puzzle) — with everything else open-ended. Pure churn until a solution.

The swarm ran five roles: a community-scouring agent (scanned public discoveries), a browser agent (submitted candidate answers to "beast bot" to harvest feedback/clues), a video agent (dissected MrBeast videos frame-by-frame for embedded clues), a puzzle solver (cracked the surfaced clues), and a coordinator that held current state and who-was-responsible-for-what — the shared-state/blackboard hub.

Which commissioning principles it embodied:

Which it lacked: there was no crisp acceptance test beyond the binary win — no intermediate done-conditions, no rubric scoring partial progress, no effort budget. That is precisely why the mode was "churn." It did not win, but it also did not collapse, mis-coordinate, or fabricate a false "solved" — because the external acceptance test made failure honest and visible. The honest "not solved" IS the lesson (Mark 4 + Mark 7): a well-commissioned task must be able to come back "not solved" truthfully. If your brief can only ever return success, you specified a metric to game, not an outcome to achieve. The improvement on a re-run would be to add intermediate acceptance criteria and an effort budget so the churn converges instead of running open-loop.

6. Mapping to our harness

Commissioning principle Where it already lives in our setup The gap to close
Specify finish line, stay method-agnostic verify-dispatch Rubric item 1 (spec specificity) flags 2+ ambiguous material decisions; the Fable memo's "commission, don't decompose" thesis Honest gap: we have no positive checklist asserting the brief states an outcome rather than a procedure. verify-dispatch catches under-specification but not over-specification (a too-prescriptive method-brief that amputates the distant-solver advantage and, on Fable, can trigger the silent Opus fallback).
Acceptance test written before work starts The "acceptance-criteria-before-output" discipline; the Fable memo's "external done-condition file" idea verify-dispatch Rubric item 6 checks output specificity (where artifacts go, return format) but does NOT require a pre-registered pass/fail acceptance test. This is the single biggest gap — a dispatch can PASS today with a clear output location and no done-condition.
Metric tracks the real goal (anti-Goodhart) /verify-strategic-output checks overconfidence-vs-evidence and cherry-picked backtests No explicit "is this metric a faithful proxy for the goal, or just the easiest thing to measure?" check in verify-dispatch.
Verify on ground the maker never saw The whole /verify-* fresh-eyes family (verify-dispatch, verify-vault-write, verify-strategic-output, design-critic, video-critic, verify-pdf-output); the "verification belongs to an independent worker" SOP Strong. The fresh-eyes-subagent-with-zero-context requirement IS the held-out verifier. Recursion stop at depth=1 is correct. Gap: none of these reserves a hidden test the producer can't see in advance — they re-judge the finished artifact, which is the right shape for soft outputs.
Audacious AND believable; bound scope verify-dispatch Rubric item 2 (scope coherence: 4+ sub-tasks -> require pre-decomp; 8+ -> split) Scope is well covered. "Believable/achievable" is not checked — a dispatch the sub-agent has no path to complete passes today.
Hold the bar; tie authority to verified increments The implementation-notes pattern (running decisions/deviations/tradeoffs file); PR-only workflow; paper-trade deploy classifier gate Implementation-notes gives the audit trail (the "process on record" the Fable memo argues for over Fable's opaque internal orchestration). Gap: no formal "milestone acceptance" event in multi-stage dispatches.
Scale verification to failure cost; HOW-rubric + gates verify-dispatch Rubric item 4 (classifier-blocking patterns: paper-trade auth, settings.json, external email, destructive git); CLAUDE.md hard rules as immutable gates; the Workflow tool's synthesis seat as the coordinator-with-shared-state Strong on irreversible/catastrophic actions. The Workflow tool IS our durable coordinator (the MrBeast coordinator role, made structural). Gap: we don't deliberately dial verification gate-count by stated failure cost — it's implicit in skill choice, not a written field in the dispatch.

The honest summary of the gap: our harness is excellent at the independent-verifier half of the canon (the /verify-* family is the held-out judge, and verify-dispatch is genuinely the pre-flight spec critic). It is weakest at the pre-registered acceptance test half — verify-dispatch does not yet require the dispatch to state a concrete done-condition, an achievability check, or a failure-cost-to-gate-count mapping. Those three are the highest-value additions, and they map cleanly onto new rubric items.

7. A dispatch checklist

Before you dispatch a sub-agent (or write any commission), write these down. Derived directly from the canon:

  1. The finish line. One sentence stating the OUTCOME, not the procedure. ("Produce X with property Y," not "do steps A, B, C.") If you can't state it in one sentence, you don't have a commission yet. (Mark 1)
  2. The acceptance test. The concrete, pre-registered pass/fail condition, checkable without arguing about it. Quantitative where possible (a number, a file that exists, a reproducible artifact); a rubric where the output is soft. Write it BEFORE the work starts. (Marks 2, 4)
  3. Does the metric track the real goal? Sanity-check the acceptance test isn't an easy-to-measure proxy that decouples from what you actually want. Encode deployability/safety/correctness INTO the test, not as an afterthought. (Mark 3)
  4. What's fixed vs open. List the constraints that are load-bearing (the box the maker works inside) and explicitly mark everything else as the maker's call. Over-specifying method amputates the distant-solver advantage; under-specifying the box invites disputes. Detailed enough not to be misinterpreted, open enough to find a better path. (Marks 1, 5)
  5. Achievable AND audacious? Is there a path the maker can see to "done"? A dispatch with no visible route is a failed dispatch. Pair with an effort budget (the brief bounds the cost of the search, not just its target — 1 agent / 3-10 calls simple, 10+ complex; reserve the swarm for high-value parallelizable work). (Mark 5)
  6. The failure cost -> how many verification gates. State how catastrophic and irreversible a miss is. Survivable + reversible -> trust the outcome, one fresh-eyes review at the end. Catastrophic or irreversible (paper-trade, channel send, commit, deploy) -> staged gates + explicit authorization + an independent verifier against the spec. Pure-binary-finish + interaction/judgment involved -> add a HOW-rubric so you don't reward the wrong behavior. (Mark 7)
  7. Who holds shared state, and is the verifier independent? For multi-agent work, name the coordinator that holds global state and assigns who-acts-next (the Workflow synthesis seat); specialists write to it, not to each other. The acceptance check must be run by someone with no stake in passing it — a fresh-eyes subagent with zero context, never the producer. Confirm an external/held-out signal exists so the task can return "not solved" truthfully. (Marks 4, 6 + the MrBeast coordinator + beast-bot signal)

If you can fill in all seven, the dispatch is ready. If you can't fill in #2, stop — you're about to adjudicate by vibes and re-dispatch endlessly.

Related