"How to design Experiments to Evaluate your Agentic Harness" — @neural_avb

Why this is in the vault

Founder shared 2026-05-10 22:05 ET in #ops Discord with no comment. Filed because this is the explicit operational methodology for harness evaluation that Osmani's harness-engineering piece (filed this morning) lacks. Osmani names the discipline (ratchet every failure into a permanent rule) but doesn't tell you HOW to test whether your ratchet actually moved the metric. AVB does. The 6-step framework is concrete, lifts cleanly, and addresses a real gap in RDCO's current /improve discipline (we ratchet on observed failures but don't run formal A/B comparisons or maintain test-case datasets per-skill).

Bookmark-to-like ratio of 2.31:1 (1306 bookmarks / 565 likes on 103k impressions) is strong practitioner-save signal — engineers are filing this as reference, not just liking it. Author has only 10.8k followers so this is travelling on substance, not reach.

The core argument (the 6 steps)

Building agentic systems = (1) Build, (2) Evaluate, (3) Refine. Step 1 is trivial with coding agents. Step 3 is trivial once you finish Step 2. Most teams underestimate Step 2.

Evaluation has two surfaces AVB explicitly distinguishes:

1. Collecting logs + success metrics on the current system (passive)
2. Validating hypotheses + comparing approaches against alternatives via deliberate experiments (active)

Most teams do (1), few do (2). The article is about (2).

Step 1 — Decide what to evaluate

Treat each agent as a separate harness. System-level vs module-level harness is a deliberate choice, not a default. Module to pick:

• Where in the pipeline have you made the most egregious assumptions?
• What runs EARLIEST in the chain? (errors propagate downstream)
• What vector are you optimizing?

Step 2 — Decide your end goal

Pick ONE optimization vector while keeping others acceptable:

• Improve accuracy
• Improve latency
• Improve cost
• Improve code quality (kill dependencies, simpler/leaner code)

Universal goal: better value to users, fast, cheap, minimal tech debt. Different businesses weight these differently. AVB's example: cost-reduction for his self-funded Paper Breakdown service.

Step 3 — Isolate the black box and your knobs

Clean function: inputs in, outputs out, no internal plumbing. Plus EXPLICIT independent variables (the knobs):

• Testing different LLMs → model name as input parameter
• Testing different prompts → prompt as input parameter
• Testing new tools → behind a feature flag

Cap at 2-3 independent variables. More = unintepretable results.

Persistence trap: turn caching/db-writes OFF for experiments. Each test case must be transactional and ephemeral. Test case 10 should have zero advantage/disadvantage from test case 4.

Step 4 — Design your test-cases

Best-to-worst sources:

• Production logs (best — actual real inputs)
• Self-written test cases (medium — biased by your assumptions)
• LLM-synthetic (worst — recursive bias risk)

If you have no production logs, "set up your analytics first" (he names PostHog) and come back later.

Quality criteria for test-case data:

• Deduplicated and diverse (avoid oversampling subdomains, biases entire experiment)
• Ground-truth responses preferred (lets you measure drift from current solution)

Step 5 — Design evaluation metrics

Deterministic > probabilistic. Deterministic metrics (string contains, valid JSON, length under N, regex on format) are cheaper, faster, 100% reliable. Use these wherever possible.

Probabilistic = LLM-as-a-judge. Use only when deterministic isn't possible. Common patterns:

• Retrieval agents → precision, recall, IOU
• Full-response agents → LLM-as-judge
• Task agents in verifiable environments → success/fail flag from environment
• Citation/coding agents → regex on output format

ALWAYS record at minimum:

• Total walltime
• Completion token usage
• Total cost
• Error rate

Step 6 — Plot results

Bar plots for response times. Scatter for cost-vs-quality / latency-vs-quality. Box plots for distribution. AVB notes you can ask your coding agent to draw plots.

Bonus — RL environments equivalence

Eval harnesses and RL environments are structurally the same: you have observations (test inputs), actions (model output), reward function (your scoring). So once you have an eval harness, you can immediately plug in prompt-optimization (GEPA) or end-to-end RL training to train smaller agents on YOUR specific tasks. This is a load-bearing aside that opens a whole capability path.

Worked example (the proof)

AVB ran his retrieval subagent eval against multiple smaller models. Result: replaced gpt-5-mini with gemini-3-flash-lite. Faster, cheaper, and a side-benefit (calls a tool the prior model missed). Now reusable: when a new model drops, re-run the same test-cases against just the new model and compare.

Mapping against Ray Data Co — STRONG

This article fills the eval-methodology gap in RDCO's harness-engineering discipline. We have the ratchet (every failure earns a rule) and the audit script (deterministic post-condition check). What we DON'T have is AVB's Step 3-6: isolated black-box per skill + curated test-case dataset + formal independent-variable A/B + plot-driven interpretation.

What we already do that maps to AVB's framework

• Step 5 deterministic-metrics — we run this via ~/.claude/scripts/audit-newsletter-outputs.py (13 invariants, zero LLM, post-condition checker). Exactly the "deterministic > probabilistic" rule. The audit's outputs feed ~/.claude/state/newsletter-audit-log.md.
• Step 6 plotting — partial. We have ~/.claude/state/eval-mine-*.json files (4 dated 2026-04-24 through 2026-05-08) but these are eval-snapshot artifacts, not plotted-and-compared visualizations. No bar/scatter/box plots being generated.
• /self-review — semantic-drift detection on a 13-criterion rubric. Closer to AVB's Step 5 LLM-as-judge pattern but with a hand-coded rubric instead of "ask another model to grade."

What we DON'T do that AVB names as load-bearing

1. Per-skill black-box isolation with independent variables. Our /improve cycle reads observed failures and edits skill prompts. We don't have a clean "skill-as-function" with feature-flagged variants we can A/B. Each /improve commit is irreversible-by-default; we can't easily compare prompt-v1 vs prompt-v2 against the same input set.
2. Per-skill test-case datasets. We have no curated tests/<skill>.csv per-skill. We have production logs (vault entries, decision logs, sub-agent traces) but they're not extracted into reusable test-case datasets.
3. Formal A/B with statistical interpretation. Every /improve change is a one-way ratchet. We never compare two skill versions against the same dataset and pick the winner with measured confidence.
4. Walltime + cost + error rate per skill run. We track sub-agent token usage at the parent-collection level but not per-skill-version with a benchmark dataset.

Implications — what /improve should learn from this

The /improve skill should evolve to support AVB's discipline. Concrete proposed changes:

• tests/ subdir per skill — ~/.claude/skills/<skill>/tests/cases.csv with input | expected_output columns sourced from the most recent N actual production runs. Default to 20 cases per skill (matches AVB's batch-of-20 newsletter pattern).
• A/B variant flag — /improve proposes a NEW prompt as a variant rather than editing in-place. Both run against the test-case dataset. Winner becomes canonical.
• Eval-mine pattern formalized — every variant comparison generates a ~/.claude/state/eval-mine-<skill>-<date>.json with metrics across the test-case dataset (mean accuracy, walltime, token usage, cost, error rate).
• Plot generation — /eval-plot <skill> skill that generates bar/scatter/box plots from the eval-mine artifacts using matplotlib (Python lift, ~30min one-time build).
• /improve becomes A/B-aware — current /improve assumes the new prompt is better. AVB's discipline says: prove it first.

This is a meaningful upgrade to the meta-loop. Worth queuing as a /improve target itself.

Implications for the RL-environments equivalence

AVB's bonus point is large for RDCO long-term. If our eval harnesses are structurally RL environments, we can:

• Run GEPA-style prompt optimization on /research-brief, /draft-review, /process-newsletter sub-agent prompts (automated prompt tuning against the test-case datasets we'd build)
• Eventually fine-tune smaller open-source models on RDCO-specific tasks using the eval-mine datasets as preference data

Both are post-Mammoth, post-healthcare-bet projects. But naming them as on-the-table possibilities means the eval-harness work pays off in capability expansion, not just hygiene.

Sanity Check candidate

Working title: "Stop ratcheting blind: the missing eval discipline in agent operating loops."

Original re-frame: most teams who hear "harness engineering" (Osmani) get the ratchet half but skip the eval half. They edit prompts based on gut + last-week's failure, then declare victory. AVB's 6-step framework is the discipline that turns "we changed the prompt" into "we proved the new prompt is better, by how much, with what tradeoffs." The Sanity Check piece would walk an operator through applying the framework to ONE skill in their stack, using RDCO's own /process-newsletter as the worked example.

Voice fit: empirical + practitioner methodology + named tradeoffs. Founder voice strength.

Tier: medium-priority candidate. Strong stand-alone but smaller wedge than the harness-moat-two-layers piece. Slot for week-of-2026-05-19 if Mammoth + healthcare-bet leave bandwidth.

Tracked-author candidate (CRM workflow)

@neural_avb (AVB). 10.8k followers, run "Neural Breakdown" on YouTube + "Paper Breakdown" SaaS for studying research papers with AI. Self-funded indie operator (the cost-reduction motivation in Step 2 is real-world signal, not academic). Engineering-first content with practitioner voice. Likely worth tracking — file as candidate for the contacts CRM (Task #4 per process-newsletter README).

Notable quotes (≤15 words each, in quotation marks)

• "Treat each agent as a separate harness."
• "Visualizations are a window to your experiment's soul."
• "Use deterministic metrics wherever possible."
• "If you don't have production logs, quit experimenting and set up your analytics first."

Open follow-ups

• Build per-skill tests/ directories with curated test-case datasets (start with /process-newsletter since it has the highest ratchet velocity)
• Build /eval-plot skill (bar/scatter/box from eval-mine JSON artifacts)
• Make /improve A/B-aware (variant prompt + dataset run + winner-takes-canonical pattern)
• Investigate GEPA prompt-optimization on RDCO sub-agent prompts (mid-term, post-test-case-dataset infra)
• Add @neural_avb to tracked-authors-candidates list

Related

• [[06-reference/2026-05-10-addy-osmani-agent-harness-engineering]] — the framework AVB's piece operationally completes (Osmani names the ratchet; AVB shows how to evaluate whether the ratchet worked)
• [[06-reference/concepts/2026-05-10-harness-moat-two-layers-portability]] — the universal-harness layer is exactly where AVB's eval methodology lives
• [[06-reference/concepts/2026-05-10-ray-architecture-introspection]] — Layer 7 (ratchet) is the layer AVB upgrades from "intuition-driven" to "experiment-driven"
• [[06-reference/research/2026-05-10-agent-harness-landscape]] — AVB's article is one more datapoint in the same-week thesis convergence (now ~6 independent operators converging on harness-engineering as the discipline)
• [[06-reference/2026-05-09-tobi-lutke-river-public-channel-agent]] — River's 36→77% merge rate improvement is exactly what AVB's framework would let you A/B-prove rather than vibes-attribute
• [[06-reference/2026-05-09-garry-tan-meta-meta-prompting-book-mirror-brain-repo]] — meta-meta-prompting needs eval discipline AVB describes; can't ratchet without measuring
• [[06-reference/concepts/]] — concept-doc candidate: "the eval-discipline gap in harness engineering"

Source caveat

Article body retrieved via xmcp getPostsById with tweet.fields: ["article", ...] + expansions: ["article.cover_media", "article.media_entities"]. Plain text returned full ~2400-word body cleanly. Six embedded media (cover + 5 illustrative diagrams/screenshots) returned as media_keys but not pulled — would need to fetch separately if Sanity Check piece requires the worked-example diagrams.

Article date 2026-03-10 (2 months old when shared); founder sharing it today on the back of the harness-engineering thesis cluster work this week. Not new content; new RELEVANCE given today's frame.