06-reference

zach lloyd warp verify then build test harness agentic coding

2026-05-12·reference·source: X (long-form article)·by Zach Lloyd
agentic-codingtest-harnessverification-loopmulti-agent-architecturemacmermaid-to-svgic-modecomplexity-ratchet

"Verify, then build" - @zachlloydtweets

Why this is in the vault

Direct lived-experience proof of the multi-agent skill build-out pipeline the founder sketched 5 hours earlier today. Lloyd shipped a multi-month engineering project (Mermaid → SVG renderer in pure Rust, no JS runtime) as a CEO in his free time by building the test harness FIRST, then running a sharded multi-agent visual-verification loop until the agent-produced renderings matched canonical references. This is the strongest evidence yet for the multi-agent pipeline decision teed up at https://hq.raydata.co/decisions/2026-05-12-multi-agent-skill-buildout-pipeline.html - converts the decision from theory to worked example with an open-source repo and published prompts.

The core argument

As code becomes cheap, correct code becomes the constraint. Sometimes the answer to that constraint is to write more code - specifically, build the test harness ahead of time, then point agents at it. If you can give agents clear success criteria they can verify against, they can autonomously build hard projects.

The methodology Lloyd demonstrates:

  1. Generate the canonical reference set first. For Mermaid: have an agent generate a huge variety of diagrams (across ~20 diagram types) and produce the canonical renderings from the reference implementation. This becomes the ground-truth oracle.
  2. Bootstrap pass. Have an agent port the reference implementation to the target stack in one shot. Produces a bad-but-functioning baseline.
  3. Visual verification loop (the load-bearing part):
    • Agent regenerates renderings using the in-progress port
    • N parallel cloud agents (Lloyd uses Warp's Oz) compare against canonical via computer vision
    • Shard the N agents by diagram type (~20 shards = ~20 parallel safe-to-work-in-isolation contexts)
    • Each agent diagnoses visual discrepancies and patches them on its own branch
    • Branches merge back to main
    • Repeat until visual differences are gone
  4. Hill climbing. Lloyd ran the loop 10-15 times over a couple of days, mostly hands-off. Each run takes about an hour and produces incremental improvement.

The key insight: the verification infrastructure is itself worth building with agents at project outset. Lloyd has used the same pattern for Markdown tables in Warp, performance-regression replication, and their multi-agent-architecture (MAA) debugger SaaS.

Lloyd's framing: "build the success validation through code at the project outset. This is what I meant when I said that sometimes the solution to writing correct code with agents is to write more code."

Open source: https://github.com/warpdotdev/mermaid-to-svg with prompts at /docs/agent-visual-verification.md.

Mapping against Ray Data Co

This is the strongest single piece of evidence for the multi-agent skill build-out pipeline the founder sketched today. Direct one-to-one mapping:

Founder's sketch (2026-05-12 iMessage) Lloyd's lived implementation
Test plan author seat Canonical-reference-set generator (independent of the coder)
Test plan code instrumentation seat The verification loop itself: regenerate + compare + diagnose
Code-writing seat The bootstrap port pass + the diagnose-and-patch agents
Build critic seat The sharded visual-verification agents (the cv-comparison fleet)
"Tests to ground truth, not tests to code" Canonical reference set is the ground truth, separate from the code being shipped

Lloyd's mermaid-to-svg case study is a worked example of the discipline. The "tests to code instead of tests to ground truth" anti-pattern the founder named is structurally prevented because the canonical reference set was generated FROM a different implementation (mermaid.js) than the one being built (the pure-Rust port). The oracle cannot be polluted by the code-under-test's choices.

Other RDCO surfaces this maps to directly:

The actionable extract for RDCO:

  1. Skill builds should produce their canonical reference set FIRST. For /build-website-discovery (currently Blocked pending the pipeline decision): the canonical set would be 5-10 known-good website-discovery artifacts from past projects (SC v3 brief, MAC brand kit, RDCO umbrella, etc.) with the documented inputs that produced them. The skill is then graded by whether new runs produce comparable artifact shape against that ground truth.
  2. Sharded parallel critics. Lloyd's 20 diagram-type shards translate to RDCO as: when reviewing a skill's output, decompose the output into orthogonal aspects (frontmatter completeness / RDCO mapping strength / sponsor disclosure / cross-link richness / etc.) and dispatch one critic-agent per aspect. Each operates in isolation. Today's /process-newsletter watch already does this for newsletter files via the audit script - but the audit is single-process. The next step is a fleet of CV-style critics where each diagnoses + patches its own aspect.
  3. Open-source the prompts. Lloyd published the visual-verification-loop prompts in the repo. RDCO's equivalent: the canonical SOP doc at 02-sops/2026-04-19-newsletter-output-invariants.md is the equivalent of his prompts file. Worth pulling in this discipline as we build more skills - SOP + invariants live next to the skill.
  4. The "CEO in free time" framing is the IC-mode pattern. Per [[../.claude/projects/-Users-ray/memory/feedback_ic_vs_production_mode]] - Lloyd is doing IC-mode hustle (CEO building Mermaid in free time) but enabled by production-grade test infrastructure. Same dichotomy applies to RDCO: founder can IC-mode pencil out a bet provided the production-mode rails are in place underneath.

Notable tensions

Related