06-reference

every ai judgment fine tuning human goals

2026-06-24·reference·source: Every·by Katie Parrott

Can AI Learn Good Judgment? — Katie Parrott (Every)

This hybrid issue of Every's Context Window digest wraps three internal experiments at Every around a common question: can AI internalize expert judgment rather than just follow rules? Plus a new AI & I podcast episode on the philosophical stakes of that question at scale.

⚠️ Sponsorship

Sponsored by Microsoft ASSERT — a tool from Microsoft's Responsible AI team that converts natural-language behavior specs into executable evaluations for AI agents, portable across dev and runtime environments. Pitched as "a behavior spec should be a first-class input to your release pipeline."

Main Essay — Dan Is Cloning Kate (But Not in a Weird Way)

Dan Shipper is attempting to fine-tune a model on 30,027+ historical edits from Every's editor-in-chief Kate Lee — not to automate editing, but to offload the tedious sentence-level copy pass so Kate can spend time on higher-order argument shaping. Earlier attempts using prompts, style guides, and skills hit a ceiling: they could encode the rules Kate could articulate, but not the judgment she applies when rules collide. The same repetition can feel lazy in one paragraph and essential in another. More instructions produced longer exception lists, not Kate-like results. The new approach changes the model itself via fine-tuning on the edit corpus.

The paywalled section covers: how the fine-tuning process is structured; the result that AI judges agreed with professional designers only 55% of the time (used as a benchmark calibration finding); and using a backlog of historically failed tasks to test whether new model versions have actually improved.

Digest Items

Turning Demonstrations into Skills (Arielle Shipper): A low-lift method for teaching agents through screen recording demonstrations — roughly two minutes of video as training signal. Frames the technique as an accessible alternative to writing explicit instructions when the workflow is easier to show than describe.

Coaching Codex Beyond Its Limits (Austin Tedesco): Approaches for prompting Codex to accomplish tasks the author couldn't do himself — specifically using Codex's web access and goal-directed autonomy to operate in unfamiliar tool environments.

Podcast: What It Will Mean to Be Human When AI Can Do Everything

Dan Shipper interviews Edwin Chen, founder of Surge AI (data environments and evals for major model labs; ~$1B revenue, no VC). Four core ideas:

  1. Saturated benchmarks. When OpenAI's model disproved an open Erdős conjecture using novel algebraic geometry, a top mathematician braced for "all over for mathematicians very soon" — then was relieved to learn the model had found a counterexample (easier) rather than proved an upper bound. Buys humanity another year or two of unique contribution at the elite level.

  2. Creation as a choice. Scaling laws suggest AI will eventually do everything better. Chen invokes Ted Chiang's framing: behave as if your decisions matter even when they don't. Making things becomes an act of will rather than capability.

  3. Agency vs. automation. AI can currently execute toward a stated goal; it cannot generate the goal itself from scratch. LLMs lack intrinsic motivation, exploration drives, or the capacity to abruptly redefine objectives. "There may be a future where AI can pursue unbounded, nebulous, completely unformed goals — but that's not happening" under current architectures. Humans still need to set the target.

  4. The engagement trap. Models trained on session length or LM Arena votes learn to "reward hack user preferences" — iterating endlessly rather than advising the user to stop. Chen spent 20 rounds on a low-stakes email until switching to Claude, which told him after a few turns to just send it. Delegation (agent acts autonomously) removes the engagement incentive; the model has no reason to keep the human glued.

Why this is in the vault

Two distinct angles map to RDCO's current work:

First, the fine-tuning-as-judgment-transfer frame is directly relevant to how RDCO's COO agent could evolve. The current harness relies heavily on CLAUDE.md rules and skill prompts — exactly the "longer instruction list" path Every already tried and found insufficient. The Dan/Kate experiment demonstrates that encoding judgment requires model-level intervention (fine-tuning on demonstrated behavior), not just better prompts. This is a concrete data point about the ceiling of the prompt-only approach.

Second, the human goals as the irreducible input argument (Edwin Chen / agency vs. automation) reinforces RDCO's current positioning: the targeting layer (what to build, for whom, against which bottleneck) is the durable human contribution. As execution becomes cheaper and more autonomous, goal-setting and target selection compound in value. This is the intellectual backing for why Ben's DSA role and RDCO's thesis-first posture (Markov phase-tracker, Sanity Check framing) isn't eroded by better models — it's amplified.

The engagement trap finding is also worth noting: RDCO's agent harness is explicitly designed toward delegation (agents execute, founder is advisory), which Chen's framing suggests is the structurally correct design for avoiding assistant-mode reward hacking.

Mapping against Ray Data Co

Reinforces existing discipline:

Surfaces a gap:

Contradicts nothing — extends the frame:

Related