06-reference

garry tan ai agent complexity ratchet 90 test coverage

2026-05-12·reference·source: Garry Tan (X long-form note)·by Garry Tan
garry-tanharness-engineering-clusterai-agent-testingtest-coveragemac-positioning-evidenceycgstackgbrain

AI Agent Complexity Ratchet (Garry Tan) — vault assessment

Why this is in the vault

This is the fourth independent harness-engineering thesis convergence point this week, and it lands from the YC-president seat with the largest practitioner footprint in the cluster. The piece is the seventh in Tan's open AI-coding series (a series the vault has been tracking since the foundational [[2026-04-11-garry-tan-thin-harness-fat-skills]] piece). Two reasons it earns immediate filing:

  1. It names test coverage as the load-bearing mechanism that turns a coding agent from a vibe-tool into a compounding system. That is the same claim MAC makes about data-quality testing for analytics agents. Garry is independently arguing for what MAC sells, in a different domain.
  2. Convergence count this week. Counting from May 5: [[2026-05-10-addy-osmani-agent-harness-engineering]], [[2026-05-10-alphasignal-self-improving-agents-harness]], [[2026-05-11-cfo-secrets-ai-for-cfos-series-synthesis]] (the Secret CFO buyer-side convergence), and now Garry Tan from the YC seat. That is four authors arguing the same thesis from four different vantage points in 7 days. The thesis is no longer "RDCO's pet idea"; it is the consensus position of practitioners shipping at scale.

The core argument

Garry's thesis in one paragraph: software engineering was built for fifty years around "prevent errors because errors are catastrophic" because human willpower could not sustain >70-80% test coverage. AI agents do not experience effort, so the brutal last 20% of coverage that used to be uneconomical is now free. At 90% coverage the defect-removal curve hits a knee where "the quality floor goes up with every turn." That one-way motion is what he calls the Agent Complexity Ratchet. He cites Capers Jones (defect-removal-efficiency knee at 85-95% coverage), DO-178C avionics MC/DC requirements, and Six Sigma reliability curves as the empirical floor for why 90% specifically is the number.

The mechanism: every agent coding session adds three things to the codebase:

  1. Tests that encode what "correct" means (regression signals)
  2. Documentation that records why decisions were made (institutional memory)
  3. Evaluation results that establish quality thresholds (cross-model scored output)

On the next turn the agent loads all three into its context window. It cannot regress below the test suite, cannot ignore the documentation, cannot ship below the evaluation baseline. Forward-only motion. That is the ratchet.

His proof of concept is GStack (93K GitHub stars, 701K lines, 46 skills) plus GBrain (14K stars). 14 PRs in 72 hours, ~29,000 net new lines, all from Claude Code + Codex via 15 simultaneous Conductor sessions, each release better-tested than the last.

Owned vocabulary / frameworks

Specific tools and frameworks named: Claude Code, Codex, Conductor (parallel agent sessions), Bun TTY harness, OpenClaw plugin runtime, GStack, GBrain. Citations: Capers Jones (Applied Software Measurement), DO-178C, Mockus/Nagappan/Dinh-Trong Windows Vista study, Andrej Karpathy ("vibecoding").

Mapping against Ray Data Co

(a) MAC framework — Scope x Basis matrix vs the 90% coverage claim

Strong overlap, different axis labels, same operational shape. MAC's framework ([[01-projects/data-quality-framework/testing-matrix-template]]) is built on two axes: Scope (column / row / aggregate) x Basis (absolute / source / production / reconciliation / temporal / human-sanity). The "definition of done" is at least one check at each scope level. Garry's "90% coverage" claim is a coverage number; MAC's "Scope x Basis matrix complete" is a structural-completeness check. These solve the same problem with two different framings:

(b) Harness-engineering thesis cluster — 4th convergence point

Strengthens the thesis materially, and adds a new dimension. The earlier convergence points (Tan's own April thin-harness-fat-skills piece, Klaassen, Pachaar, Osmani, AlphaSignal, Secret CFO) all argued for the harness-skills architecture. Garry's new piece argues for the quality-floor mechanism that makes harness-skills compoundable over time. A thin harness with fat skills is not enough; without the ratchet, the skills regress whenever the model version changes or a new contributor opens a PR.

This is a new dimension. Prior cluster docs framed the harness as a spatial architecture (where intelligence lives). The ratchet frames it as a temporal architecture (how quality compounds across turns). Both matter. The cluster vocabulary should now include "ratchet" alongside "harness," "skills," and "resolvers."

(c) Specific MAC content angle

Yes - file a candidate angle. Working title: "Garry Tan says 90% coverage is required. Here is the matrix that gets you there in analytics." The piece writes itself:

  1. Open with Garry's claim and the YC-president source credibility
  2. Concede the claim is correct for code
  3. Pivot: data pipelines fail in a category his 90%-of-lines never catches (aggregate-level silent drops)
  4. Present the Scope x Basis matrix as the analytics-domain version of the same insight
  5. Close with the "ratchet" framing applied to the matrix: every analytics PR adds one new test in each scope-basis cell, the matrix can only get more complete

This is exactly the kind of "Sanity Check needs original re-frame" piece the founder demands (per memory). We are not restating Garry's claim; we are extending it into a domain where his coverage number undercounts the real problem.

(d) Validates or contradicts Thin-Harness-Fat-Skills?

Validates and extends. The April thin-harness-fat-skills piece described the architecture. This new piece describes the mechanism that lets the architecture compound over time. No contradiction; the ratchet sits on top of the harness-skills split. In the three-layer architecture (fat skills / thin harness / deterministic application), the ratchet is what guarantees the fat-skill layer accumulates quality monotonically rather than regressing when models change or contributors land PRs. The two pieces should be read as one combined position: thin harness + fat skills + 90% coverage ratchet = compounding agent system.

Comparison to the Secret CFO's MAC validation (yesterday)

Both seats now argue for test/verification rigor as the load-bearing constraint on AI agents, from different vantage points. The convergence is striking:

Dimension Garry Tan (May 12) Secret CFO (CFO Secrets, May 11 synthesis)
Seat YC president, builder, OSS maintainer Anonymous PE/PCo CFO, buyer-side, finance ops
Surface AI coding agents on a codebase AI agents touching finance/books-of-record
Mechanism named Agent Complexity Ratchet (tests + docs + evals) Deterministic vs probabilistic decomposition; hub-and-spoke governance
Threshold 90% test coverage, every PR, no exceptions "Don't let LLM reasoning touch master data or payment rails"
Cost frame Effort wall demolished by agent labor Token cost as new governance line, Moonshot Pot capital model
Conclusion Verification rigor is now free; ship at 90% Verification rigor is non-negotiable; route by reasoning type

Same conclusion from two different vantage points. Builder side says "verification used to be too expensive, agents made it free, so 90% is the new floor." Buyer side says "AI without verification near books-of-record is unacceptable." Different framings, same load-bearing claim: the harness must include a verification substrate, and the substrate is now economically tractable. This is the cleanest two-sided validation of the MAC thesis the vault has captured.

Related