Airbnb — Eval-Driven Development (GenAI at scale)

Founder shared 2026-08-03 (not caught by the newsletter watch — Airbnb Engineering's Medium is not a whitelisted sender). Assessed via subagent extract; verdict delivered in-channel: skim, don't read — 70-80% restates standard eval wisdom (Husain/Shankar/Yan territory), but the specifics below are keepers.

What's actually new here

• Judge calibration protocol with hard numbers. 50-100 example golden set (deliberately seeded with bad examples), agreement measured via Cohen's kappa / Krippendorff's alpha, acceptance bar in the high 80s-90s% human agreement, periodic recalibration. Worked example: a faithfulness judge at 78% agreement was found to penalize accurate paraphrases; one rubric fix took it to 88%. Most eval writing says "calibrate your judge" without ever stating the threshold or the statistic.
• The experts-disagree gate. Start human eval with 20-100 SME-labeled rows; if the humans disagree with each other, stop and fix the rubric before automating. Don't scale human disagreement into a judge.
• One judge per dimension — explicitly no "God evaluator" scoring everything at once.
• Trajectory evals for agents: score individual tool calls and intermediate states via trace reconstruction (DFS over the agent tree), not just the final answer.
• Same-metrics-in-prod: production monitoring samples ~5% of daily de-identified traffic through the same checks and judges used pre-production — one eval system, not parallel ones.

The standard 70-80% (for completeness)

Five principles (goals upfront · real errors drive metrics · small evaluator sets · one human tiebreaker · product-partner loop); three-layer stack (programmatic checks → calibrated LLM-judges → human ground truth); "when in doubt, look at your data" (manually review ~100 examples, build evals from observed failure modes, never from a metric menu). End-to-end example: support-QA bot, 100-input review → 15 fabrications / 8 verbose / 5 over-refusals / 3 broken JSON → one eval per failure mode → 60-example golden set → 5,000 scaled.

Synthesis for RDCO

1. Skill retune (live now): before trusting any LLM-judge comparing original-vs-staged skill outputs, calibrate it against founder verdicts on a small golden set and require the high-80s-to-90s agreement bar. The current comparator runs uncalibrated; the check-board eval's "variance not text" confusion (2026-08-02, session-internal, unfiled) is exactly the failure this protocol catches. Same design lesson for station-critic: the one-judge-per-dimension rule maps 1:1 onto its existing critic-axis fan-out, external validation of that design plus the calibration step it currently lacks.
2. phData FDE work: big-brand-endorsed framework hand-able to enterprise clients nearly as-is in agent design/build engagements (five principles + three-layer stack + kappa calibration). The trajectory-eval section is direct ammo for agent-build scoping.
3. Sanity Check: NOT publishable as a summary (derivative — [[feedback_no_derivative_sanity_check_pieces]] applies). Only original re-frame seed: evals as an org-alignment problem, not a testing problem. "Solve human disagreement before you automate it," with Airbnb's 78→88 anecdote as evidence, not topic.

Related

• [[2026-07-24-thariq-context-engineering-claude-5-rules]] — the skill-retune source guidance this calibration bar should attach to
• [[2026-07-28-seeded-defect-benchmark-preregistration]] — same pre-registered-bar discipline, applied to model comparison
• [[2026-04-15-thariq-claude-code-session-management-1m-context]] — the context-rot guidance behind the subagent-extract route this assessment itself used