06-reference

lotte verheyden evals explained langfuse academy

2026-05-20·reference·source: X (Twitter long-form)·by Lotte Verheyden (@lotte_verheyden)
ai-evalsllm-evaluationverificationlangfuse-academyfresh-eyes-patternmanual-review-firstbinary-scoreseval-rubric

Evals, explained (Lotte Verheyden, Langfuse Academy)

Why this is in the vault

Clean operational rubric for AI evaluation — manual review, then code-based checks, then LLM-as-judge — published as part of Langfuse Academy's AI engineering lifecycle series. It directly maps onto RDCO's verify-* fresh-eyes subagent stack and gives us an external frame for whether the architecture is sound. Signal-to-noise on Lotte's account is high (815 bookmarks vs 389 likes on this post — bookmarked-as-reference more than performatively liked).

The core argument

Offline evaluation is the step between running an experiment and shipping. Most teams evolve in a fixed order: start by manually reviewing outputs to build intuition for what good and bad look like, then identify specific failure modes, then automate. "Teams that skip this step and jump straight to automated evaluation often end up measuring things that don't matter." Manual review is not a one-and-done; mature production setups keep continuous human review to catch new failure modes and keep automated evaluators calibrated.

Three evaluation methods, each suited to different checks. Manual builds the ground-truth labels and the intuition that tells you which automated evaluators to build. Code-based evaluators handle deterministic checks (valid JSON, schema conformance, keyword presence, length limit, SQL executes) — fast, cheap, repeatable, but cannot assess meaning. LLM-as-judge handles qualities that require understanding language (relevance, tone, summary fidelity) — necessary but imperfect, prone to blind spots especially when the judge and the application share a model family. Mature setups use all three.

Two design recommendations stand out. First, "prefer binary scores (pass/fail) over graded scales (1-5)." Graded scales introduce ambiguity about what a 3 means versus a 4 and degrade consistency across evaluators and time. Binary forces a clear definition of acceptable vs unacceptable. Second, generic qualities like "helpfulness" or "quality" rarely produce useful signal — the more precisely you can define what good and bad look like, the more useful the evaluator. Each quality you care about gets its own evaluator.

She also splits reference-based vs reference-free evaluators. Reference-free can apply to unseen production data, which makes them the right shape for live-traffic monitoring after deployment. The loop closes when production-safe checks confirm that real-traffic quality matches what you saw offline; mismatches feed back into the dataset for the next experiment cycle.

Mapping against Ray Data Co

(a) RDCO's verify- fresh-eyes stack maps directly to her rubric.* [[2026-05-19-verification-as-independent-worker-pattern]] formalizes verify-vault-write, verify-strategic-output, verify-dispatch, video-critic, and design-critic as code-callable critic agents. In Lotte's framing, we are operating at the "code-based + LLM-as-judge" maturity tier — the critics ARE structured LLM judges with deterministic post-condition checks baked in. Her warning lands hard though: we are not running periodic manual spot-checks on the critics themselves. If a verify-* subagent silently drifts into measuring something that doesn't matter, we will not catch it. Action: schedule a recurring manual-review pass over a sample of critic verdicts to confirm calibration. This is the missing tier.

(b) Binary-over-graded recommendation maps to PASS/ITERATE/SCRAP — settled 2026-05-20 as a two-gate reframe. Founder's reframe in response to this finding: two gates instead of three states. Gate 1 = pass/fail (binary, decisive). Gate 2 (only if Gate 1 = fail) = iterate (close, cleanup tweaks) vs scrap (dead end / fresh-eyes reset). Same 3 outcomes downstream, but the critic's cognitive load shrinks to one binary call plus one conditional binary. Architecture decision filed at [[../02-sops/2026-05-20-verify-stack-two-gate-pass-fail-architecture]]; rollout across all 6 verify-* + critic skill files is queued as a single-batch follow-up so the surface stays consistent. Lotte's binary-over-graded warning is satisfied: each gate is binary; the second gate only fires when the first failed.

(c) Audit-newsletter-outputs.py is the canonical "code-based eval" tier. The 13 deterministic post-condition checks run with zero LLM calls, which is exactly what [[2026-04-19-kingsbury-future-of-everything-is-lies]] warned against losing — verification logic that doesn't itself depend on the thing being verified. Lotte's rubric validates this architectural choice. The Kingsbury critique of LLM-verifying-LLM contamination is partially addressed by keeping the deterministic tier outside the model loop. Same intuition appears in [[2026-04-20-every-ai-autopilot-verification-decay]] (verification decay in agent loops) and in [[2026-05-19-cloudflare-cyber-frontier-models]] (Glasswing harness with explicit eval/validate stages — parallel architectural pattern at production scale).

Tactics worth borrowing

  1. Continuous manual review as a scheduled ritual, not a phase. Add a recurring cron — sample 5 critic verdicts per week, founder spot-checks for drift. This is the missing manual-tier calibration loop.
  2. Reference-free evaluators on live traffic. Most RDCO critics are reference-based (verdict vs storyboard, vault note vs rubric). Worth identifying which checks could run reference-free against production output — e.g., a daily scan of newsletter drafts against AI-tell patterns (em-dashes, declarative-essay cadence) without needing a paired reference.
  3. Failure-mode catalog before more critics. Before building the next verify-* skill, write down the specific failure modes it would catch. If we cannot define them precisely, the critic will measure vague things vaguely.

Related