NBER w35275 verified: the 741%/20% pairing is verbatim in the paper, but it is the superseded May 2026 version and the unit is lines changed, not "code generated"
The question
Verbatim: "What does NBER working paper w35275 (Writing Code vs. Shipping Code, MIT + Wharton) actually measure and find? Pull the paper itself rather than the AlphaSignal relay."
This is a number-verification brief, not topic exposition. The vault has held generation +741% against releases +20% as the load-bearing quantity in the verification-ceiling argument, sourced only through an AlphaSignal newsletter gloss.
Verdict
CONFIRMED AS TO THE ORIGINAL, SUPERSEDED AS TO THE CURRENT VERSION. Three separate findings:
The paper, number, title, and institutions all check out exactly. NBER Working Paper 35275, "Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools," by Mert Demirer (MIT Department of Economics and NBER), Leon Musolff (Wharton School, University of Pennsylvania, Business Economics and Public Policy, and NBER), and Liyuan Yang (MIT). Dated May 2026, revised September 2026. "MIT + Wharton" is accurate.
Both relayed figures are verbatim in the May 2026 original. Original Table 5 ("Attenuation of Productivity Effects Across Production Layers"), Sync Effect row: Lines Changed 741.3% (SE 19.2), Releases 20.3% (SE 2.6). The original's own prose: "Sync agents lead to a 741% increase in lines of code and a 65% increase in pull requests, yet releases rise by only 20%." AlphaSignal relayed accurately and so did the vault note. This is not a misattribution.
The September 2026 revision restated the generation figure upward by 29% and left releases flat. Current Table 6, Sync Effect row: Lines Changed 957.5% (SE 23.2), Pull Requests 86.0%, Releases 19.8% (SE 2.9). The current prose reads "sync agents lead to a 958% increase in lines of code and an 86% increase in pull requests, but releases rise by only 20%." So the correct citation today is 958%/20%, not 741%/20% — the vault figure is stale by one revision, not wrong.
No paywall. Both PDFs downloaded freely over plain HTTPS. Confusing detail worth recording: the file at /system/files/working_papers/w35275/w35275.pdf is the current September 2026 revision, while /revisions/w35275.rev0.pdf is the original May 2026 version. "rev0" means revision zero, i.e. the oldest, which inverts the intuitive reading. The public nber.org abstract page carries the current revision's abstract.
What we already know (from the vault)
- [[2026-06-07-alphasignal-async-agents-release-bottleneck]] is the single source of the figure. Its claim: "sync agents drove a 741% increase in raw lines of code and +65% PRs, but only a ~20% increase in actual releases; async agents boost PR creation +71.8% but cannot release directly — a human must review and merge." It also recorded cumulative commits 180% attenuating to ~50% at project level and ~30% at releases, elasticity ~0.25, and "~100k GitHub devs."
- That note did the right thing: it already identified the underlying primary source by number and full title, and flagged it as "worth pulling directly if this thesis gets formalized." The gap was that nobody pulled it.
- [[2026-10-08-verification-ratio-senior-signoff-throughput]] calls this the most load-bearing quantity the vault holds on the size of the verification choke, while sourcing it only to the newsletter relay.
- [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] measured a 2-in-10 finding-level defect rate on exactly this citation shape (secondhand of secondhand through a newsletter gloss), flagging this one as elevated risk.
What the web says
- What the paper actually measures. Developer-level matched event study on GitHub commit and pull-request histories from January 2020 through May 2026 (388 million commits), joined to GitHub Copilot usage telemetry at the individual level for April 2025 through December 2025, plus a panel of four software marketplaces from March 2020 through August 2026. Current version: "more than 500,000 GitHub developers." Original version: "more than 100,000" — the sample roughly quintupled in revision, which is the likely driver of every number moving. (NBER w35275, abstract and Section 2)
- The unit behind "741%" is narrow. It is the event-study coefficient on weekly total lines changed, 1% winsorized, for the weeks 21-30 post-adoption bin, normalized by the pre-period mean, for sync-agent adopters pooled across Claude Code, GitHub sync, and OpenAI Codex, against matched non-adopting developers. The "20%" is weekly repository release count for the same group. It is not "code generated" in any general sense and not an organization-level measurement. (Current Table 6 notes)
- The commit-chain numbers also moved between versions. Current abstract: autocomplete, interactive (sync) agents, and autonomous (async) agents raise commits by cumulative 30%, 180%, 240%; the 240% falls to 80% for number of projects and 30% for actual releases; elasticity of substitution 0.23. Original abstract: 40%, 140%, 180%; 180% falls to 50% and 30%; elasticity 0.25. The vault's "180% cumulative" was the original's async figure; in the current version 180% is the sync cumulative figure. (NBER w35275, both abstracts)
- Claude Code is the outlier, and the only tool with a real release effect. Current Table 6 by tool: Claude Code Lines Changed 1343.4%, commits 295.5%, PRs 178.9%, releases 39.4% (SE 6.8). GitHub sync 935.8% / releases 17.3%. OpenAI Codex 777.8% / releases 5.2% with SE 5.5 — indistinguishable from zero. Autocomplete's release effect is 9.0% with SE 5.0, also weak. The release gain is concentrated in one tool.
- Async agents have no release estimate at all. The paper states releases are not reported for async tools "because releases are measured at the repository level and cannot be split into agent- and human-authored components." The newsletter's "async agents cannot release directly, a human must review and merge" is an interpretive gloss on a measurement limitation, not a paper finding.
- The paper's mechanism is a calibrated model, not a measured review step. Attenuation is explained by a hierarchical production model whose key parameter is the elasticity of substitution between upstream output and downstream human effort, calibrated to 0.23. The paper attributes direct bottleneck evidence, specifically longer review times, to other work (Chen and Stratton 2026), not to its own estimates.
- Scope caveat the authors state themselves. "Our data cover only open-source software and four major software marketplaces... our data have little coverage of enterprise software." They note other research finds the same attenuation pattern in enterprise software "with effects considerably smaller than ours," reasoning that open-source developers work in small teams with less organizational friction.
- Marketplace result. Current abstract: a sharp increase in new apps but no increase in total usage. The original said a "moderate" increase. Closing line: "Large task-level AI productivity gains have therefore translated only partially into shipped and used software thus far."
- The authors pre-empted the obvious objection to a lines-of-code numerator. They report a robustness check that "neither a decline in the quality of commits nor an increase in the verbosity of AI-generated code accounts for" the productivity effects or the attenuation.
- Disclosure, for the record. Demirer and Musolff previously held postdoctoral positions at Microsoft and are paid research consultants for the company. The Copilot telemetry is Microsoft data. Funding in part from Wharton's Mack Institute for Innovation Management and the Chicago AI Incubator. Not peer-reviewed; NBER working papers carry an explicit no-peer-review notice.
Convergences and contradictions
- Converges on substance, drifts on digits. The vault's qualitative claim, that generation gains attenuate hard and releases barely move, is exactly what the paper argues and is unchanged across both versions. Every specific number the vault carried has since moved: 741 to 958, 100k to 500k devs, 0.25 to 0.23 elasticity, and the commit-chain figures reshuffled across tool generations.
- The pairing was chosen to maximize the apparent choke. 741/20 is a 37x attenuation ratio because it pairs the most inflated available numerator (lines changed) against releases. The same Sync Effect row offers commits: 109%/20% in the original, 153%/20% in the current, a 5-8x ratio. Both are real numbers from the same row of the same table. The verification-ceiling argument is roughly 5x weaker if stated in commits and roughly 4x weaker if stated in pull requests (86%/20%). Nothing in the vault records that the unit choice was doing this much work.
- The vault's framing over-reads the mechanism. [[2026-10-08-verification-ratio-senior-signoff-throughput]] treats this paper as evidence about human review capacity specifically. The paper's evidence is a reduced-form attenuation pattern plus a structural elasticity; it does not measure review time, reviewer fatigue, or sign-off throughput. It is strong evidence that something human-shaped binds downstream, and it names review as the candidate, but it did not measure review.
- Contradiction with the vault's population assumption. The choke this paper measures is in open-source repositories, where the authors argue organizational friction is lower than enterprise. They cite enterprise effects as "considerably smaller." For an argument aimed at enterprises shipping agents to production, the enterprise-relevant attenuation is likely steeper on the generation side and the open-source numbers are the optimistic case, not the conservative one.
Synthesis for RDCO
The headline is good news with a maintenance bill attached. The founder's most-cited external number is real, it is in a credible NBER working paper from MIT and Wharton economists, it is reproducible to a specific table row, and the paper is free to download. "Agents in production" positioning that leans on this paper is standing on a real foundation. That is not the usual outcome of a number-verification brief, and it is worth banking: the June 7 AlphaSignal note did its job correctly, including naming the primary source by working-paper number so this check was cheap to run four months later.
The maintenance bill is that the figure is stale by one revision and the vault has no mechanism that would have caught it. The September 2026 revision roughly quintupled the sample and moved the generation coefficient from 741% to 958%. Citing 741% today is citing a withdrawn number from a paper whose current version says 958%. The fix is mechanical: update the citation to 958% lines changed against 20% releases (NBER w35275, September 2026 revision, Table 6) everywhere the pairing appears, and record that working papers revise. The deeper lesson is that a number pulled from a preprint has a version, and the vault currently stores numbers without versions. Anything cited from arXiv, SSRN, or NBER needs a version or revision-date stamp alongside the figure, or the same drift recurs silently.
The substantive correction is the unit, and it cuts against RDCO's framing rather than for it. "Generation up 741%" reads as "AI produces 7.4x more work product." What the paper measured is weekly lines changed, which is the single metric most inflated by agentic tools and the one least connected to anything a reviewer must actually approve. The same table row, in the same regression, says commits rose 153% and pull requests rose 86% against releases at 20%. A reviewer's queue is pull requests, not lines. Stating the choke as 86% in, 20% out is both more honest and more defensible, and it is still a large attenuation. The 37x version invites exactly the objection a skeptical CTO will raise, that lines of code is a junk denominator in the agent era, and the authors' own verbosity robustness check is about the attenuation pattern rather than about lines-of-code being a good productivity measure. Switching to the pull-request framing costs rhetorical punch and buys the argument immunity to the one attack that would land.
Two further items change how the paper should be used. First, the per-tool breakdown is the most RDCO-relevant number in the paper and the vault does not have it: Claude Code shows a 39.4% release effect, roughly double the pooled sync figure, while OpenAI Codex shows 5.2% with a standard error of 5.5, statistically indistinguishable from zero. Tool choice appears to determine whether generation gains reach production at all. For a credibility domain built on agents actually shipping, that finding is more differentiated and more useful than the aggregate choke, and it is a genuinely original hook rather than a restatement of the newsletter. Second, the scope caveat should be stated rather than hidden: this is open-source data with, in the authors' words, "little coverage of enterprise software," and they expect enterprise effects to be smaller. Leading with that caveat is the credibility-building move, because an audience that discovers it independently discounts everything else said alongside it.
Why this is in the vault
This closes the open primary-source check on the one external figure the verification-ceiling argument rests on, and it changes two specific citations: [[2026-10-08-verification-ratio-senior-signoff-throughput]] and [[2026-06-07-alphasignal-async-agents-release-bottleneck]] both need the 741% updated to 958% with a revision stamp, and any public-facing use of the pairing should switch the numerator from lines changed to pull requests before it goes into a talk, post, or phData sales conversation.
Open follow-ups
- Chen and Stratton (2026) is the enterprise-software analogue the authors cite for "the same attenuation pattern, with code review as the binding constraint" and "effects considerably smaller than ours." That paper is closer to RDCO's actual audience than w35275 is, and the vault has nothing on it. Pull it.
- Does the vault contain other figures taken from preprints without a version stamp? A sweep of 06-reference for arXiv, SSRN, and NBER citations, checking each against the current version, would size the drift problem rather than guess at it.
- Cui et al. (2026) is the field experiment the authors benchmark their autocomplete estimates against. A randomized comparison point for the same tool in the same period is a stronger citation than an observational event study for any claim about autocomplete.
- The paper's elasticity of substitution of 0.23 is a calibrated structural parameter, not a measurement. What would it take to estimate the equivalent parameter for RDCO's own agent pipeline, where the reviewer is a single person? That is the number the operating model actually needs and the paper does not supply.
- The marketplace result, new apps up sharply with total usage flat, is a distribution and demand finding rather than a verification finding, and the vault has not engaged with it. If generation gains stop at usage rather than at review, the binding constraint is attention, not sign-off, which would redirect the positioning.
Related
- [[2026-06-07-alphasignal-async-agents-release-bottleneck]] - the relay this brief verified; needs the 741 to 958 update and a revision stamp
- [[2026-10-08-verification-ratio-senior-signoff-throughput]] - the brief that named this the most load-bearing quantity in the verification-choke argument
- [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] - the companion audit that flagged this citation shape as elevated-risk; this check is a partial vindication of the method and a partial miss on the defect type, which was staleness rather than misattribution
- [[2026-05-09-data-engineering-central-cognitive-overload-ai-development]] - the reviewer-fatigue thesis the paper is often read as supporting but does not directly measure
- [[2026-06-09-alphasignal-frontiercode-models-13-of-100]] - adjacent evidence on the gap between generated code and code that survives real-world standards
Sources
- NBER Working Paper 35275, Mert Demirer, Leon Musolff, Liyuan Yang, "Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools," May 2026, revised September 2026. Abstract page: https://www.nber.org/papers/w35275
- Current September 2026 revision PDF (read in full, not paywalled): https://www.nber.org/system/files/working_papers/w35275/w35275.pdf — Table 6, abstract, Section 1, Section 2, Section 5
- Original May 2026 version PDF (read in full; note "rev0" denotes the oldest version): https://www.nber.org/system/files/working_papers/w35275/revisions/w35275.rev0.pdf — Table 5, abstract, Section 1
- ~/rdco-vault/06-reference/2026-06-07-alphasignal-async-agents-release-bottleneck.md
- ~/rdco-vault/06-reference/research/2026-10-08-verification-ratio-senior-signoff-throughput.md
- ~/rdco-vault/06-reference/research/2026-10-08-finding-level-citation-audit-arxiv-gloss.md