06-reference/research

nber w35275 writing vs shipping code

2026-10-15·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
verification-ceilingagents-in-productionnumber-verificationai-productivitynber

NBER w35275 verified: the 741%/20% pairing is verbatim in the paper, but it is the superseded May 2026 version and the unit is lines changed, not "code generated"

The question

Verbatim: "What does NBER working paper w35275 (Writing Code vs. Shipping Code, MIT + Wharton) actually measure and find? Pull the paper itself rather than the AlphaSignal relay."

This is a number-verification brief, not topic exposition. The vault has held generation +741% against releases +20% as the load-bearing quantity in the verification-ceiling argument, sourced only through an AlphaSignal newsletter gloss.

Verdict

CONFIRMED AS TO THE ORIGINAL, SUPERSEDED AS TO THE CURRENT VERSION. Three separate findings:

  1. The paper, number, title, and institutions all check out exactly. NBER Working Paper 35275, "Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools," by Mert Demirer (MIT Department of Economics and NBER), Leon Musolff (Wharton School, University of Pennsylvania, Business Economics and Public Policy, and NBER), and Liyuan Yang (MIT). Dated May 2026, revised September 2026. "MIT + Wharton" is accurate.

  2. Both relayed figures are verbatim in the May 2026 original. Original Table 5 ("Attenuation of Productivity Effects Across Production Layers"), Sync Effect row: Lines Changed 741.3% (SE 19.2), Releases 20.3% (SE 2.6). The original's own prose: "Sync agents lead to a 741% increase in lines of code and a 65% increase in pull requests, yet releases rise by only 20%." AlphaSignal relayed accurately and so did the vault note. This is not a misattribution.

  3. The September 2026 revision restated the generation figure upward by 29% and left releases flat. Current Table 6, Sync Effect row: Lines Changed 957.5% (SE 23.2), Pull Requests 86.0%, Releases 19.8% (SE 2.9). The current prose reads "sync agents lead to a 958% increase in lines of code and an 86% increase in pull requests, but releases rise by only 20%." So the correct citation today is 958%/20%, not 741%/20% — the vault figure is stale by one revision, not wrong.

No paywall. Both PDFs downloaded freely over plain HTTPS. Confusing detail worth recording: the file at /system/files/working_papers/w35275/w35275.pdf is the current September 2026 revision, while /revisions/w35275.rev0.pdf is the original May 2026 version. "rev0" means revision zero, i.e. the oldest, which inverts the intuitive reading. The public nber.org abstract page carries the current revision's abstract.

What we already know (from the vault)

What the web says

Convergences and contradictions

Synthesis for RDCO

The headline is good news with a maintenance bill attached. The founder's most-cited external number is real, it is in a credible NBER working paper from MIT and Wharton economists, it is reproducible to a specific table row, and the paper is free to download. "Agents in production" positioning that leans on this paper is standing on a real foundation. That is not the usual outcome of a number-verification brief, and it is worth banking: the June 7 AlphaSignal note did its job correctly, including naming the primary source by working-paper number so this check was cheap to run four months later.

The maintenance bill is that the figure is stale by one revision and the vault has no mechanism that would have caught it. The September 2026 revision roughly quintupled the sample and moved the generation coefficient from 741% to 958%. Citing 741% today is citing a withdrawn number from a paper whose current version says 958%. The fix is mechanical: update the citation to 958% lines changed against 20% releases (NBER w35275, September 2026 revision, Table 6) everywhere the pairing appears, and record that working papers revise. The deeper lesson is that a number pulled from a preprint has a version, and the vault currently stores numbers without versions. Anything cited from arXiv, SSRN, or NBER needs a version or revision-date stamp alongside the figure, or the same drift recurs silently.

The substantive correction is the unit, and it cuts against RDCO's framing rather than for it. "Generation up 741%" reads as "AI produces 7.4x more work product." What the paper measured is weekly lines changed, which is the single metric most inflated by agentic tools and the one least connected to anything a reviewer must actually approve. The same table row, in the same regression, says commits rose 153% and pull requests rose 86% against releases at 20%. A reviewer's queue is pull requests, not lines. Stating the choke as 86% in, 20% out is both more honest and more defensible, and it is still a large attenuation. The 37x version invites exactly the objection a skeptical CTO will raise, that lines of code is a junk denominator in the agent era, and the authors' own verbosity robustness check is about the attenuation pattern rather than about lines-of-code being a good productivity measure. Switching to the pull-request framing costs rhetorical punch and buys the argument immunity to the one attack that would land.

Two further items change how the paper should be used. First, the per-tool breakdown is the most RDCO-relevant number in the paper and the vault does not have it: Claude Code shows a 39.4% release effect, roughly double the pooled sync figure, while OpenAI Codex shows 5.2% with a standard error of 5.5, statistically indistinguishable from zero. Tool choice appears to determine whether generation gains reach production at all. For a credibility domain built on agents actually shipping, that finding is more differentiated and more useful than the aggregate choke, and it is a genuinely original hook rather than a restatement of the newsletter. Second, the scope caveat should be stated rather than hidden: this is open-source data with, in the authors' words, "little coverage of enterprise software," and they expect enterprise effects to be smaller. Leading with that caveat is the credibility-building move, because an audience that discovers it independently discounts everything else said alongside it.

Why this is in the vault

This closes the open primary-source check on the one external figure the verification-ceiling argument rests on, and it changes two specific citations: [[2026-10-08-verification-ratio-senior-signoff-throughput]] and [[2026-06-07-alphasignal-async-agents-release-bottleneck]] both need the 741% updated to 958% with a revision stamp, and any public-facing use of the pairing should switch the numerator from lines changed to pull requests before it goes into a talk, post, or phData sales conversation.

Open follow-ups

Related

Sources