Hylak's "How to Eval" — floor-raising over benchmark-maxxing
Why this is in the vault
Founder flagged 2026-05-28 as "likely a good reference." Verdict on assessment: READ, selectively crib for our verification SOPs, don't treat as canonical. Frame is sound; quant rigor is weak; underlying marketing funnel for Raindrop is unsubtle.
Sponsorship
Self-funnel. Hylak founded Raindrop (paid LLM eval platform + OSS local tool); the guide is structurally top-of-funnel for that product. Every architectural recommendation lands on "don't use hosted dashboards" (i.e. don't use LangSmith/Braintrust competitors) and "use a local-first tool" (i.e. Raindrop's OSS tier with paid upsell). No third-party sponsor; the bias is straight self-consulting. Disclosed implicitly via Raindrop branding throughout, never explicitly called out as advertorial.
The core frame
Hylak's central move: stop chasing benchmark scores ("benchmark-maxxing"), instead do floor-raising error-analysis — read real failures from production traffic, build a taxonomy of recurring failure modes, fix the ones that matter. Eval is detective work on a small corpus of real bad outputs, not a Kaggle competition.
Published May 2026. Author = Ben Hylak, founder of Raindrop (paid eval platform + an OSS local tool). The guide doubles as a top-of-funnel for Raindrop, so every architectural recommendation lands on "don't use hosted dashboards like LangSmith/Braintrust" — discount accordingly.
The meat (what's worth cribbing into RDCO)
- Stumbles → Issues → Signals → Experiments volume-scaling ladder. As eval volume grows, the workflow has to graduate. We're at the Stumbles tier (read raw logs, build the taxonomy by hand) on cron-driven skills today. Concrete graduation path before we automate self-review — useful for sequencing the verification roadmap.
- "Prune ruthlessly — if an eval hasn't failed in 3 months, question it." Our
/verify-*fresh-eyes pattern accumulates checks; we have no retirement discipline. Worth folding into the [[2026-05-19-verification-as-independent-worker-pattern]] SOP as an explicit step. - "Reconstruct the run, ask the agent what happened." Cheaper alternative to spawning a separate fresh-eyes sub-agent for some verification cases. Worth A/B testing against
/verify-vault-writeand/verify-strategic-output— when does self-reconstruction match a fresh-eyes pass, and when doesn't it? - Code-aware / harness-backed evals over prompt-only grading. Validates the existing bias — sub-agent dispatch with real tool calls beats isolated prompt grading. No architecture change needed; this is confirmation.
The discount (where it falls short for us)
- Zero on quantitative/backtest discipline. Explicitly out of scope. The Markov capital-cycle pipeline [[2026-05-27-markov-equities-pipeline-spec]] gets nothing here — walk-forward, calibration, leave-one-out remain owned by
/investing:backtest-thesis. - Statistical rigor is hand-wavy. One A/B example cites p<0.001 with no discussion of sample size, power, or multiple-comparison correction. Qualitative-first framework, not quant-rigorous. Treat the frame as load-bearing; treat the numbers as illustrative.
- Marketing-funnel shape. "Don't use hosted dashboards" maps suspiciously cleanly onto "use Raindrop instead." Take the floor-raising frame; ignore the tool recommendation.
- No sub-agent fan-out or context-management coverage. The two things most relevant to how we actually run agent loops — absent.
Mapping against Ray Data Co
| Hylak claim | Our state | Action |
|---|---|---|
| Read real failures, build a taxonomy | Doing it ad hoc via /self-review |
Formalize a failure-mode taxonomy for cron-driven skills |
| Prune evals that haven't failed in 3mo | Our verification SOPs only grow | Add retirement-pass step to [[2026-05-19-verification-as-independent-worker-pattern]] |
| Self-reconstruct vs fresh-eyes | Always-fresh-eyes per [[feedback_fresh_eyes_subagent_for_own_artifacts]] | A/B test self-reconstruction on a sample of /verify-vault-write cases |
| Code-aware over prompt-only | Already our bias (sub-agent dispatch w/ tools) | No change |
Related
- [[2026-05-19-verification-as-independent-worker-pattern]] — our umbrella verification SOP; "prune ruthlessly" + self-reconstruction A/B are inputs to this
- [[feedback_fresh_eyes_subagent_for_own_artifacts]] — the always-fresh-eyes bias the self-reconstruction idea challenges
- [[2026-04-15-thariq-claude-code-session-management-1m-context]] — Anthropic's context-rot guidance; our reason for keeping verification sub-agent-scoped
- [[feedback_implementation_notes_sub_agent_pattern]] — implementation-notes file for the Markov build; eval framework here is the wrong tool for that quant-discipline gap