The frequency half is arithmetic, the "beats the highest-stakes one" half has no measured support, and the formal result makes frequency and stakes multiply rather than compete
The question
"Is there measured evidence that instrumenting the most FREQUENT decision beats the highest-STAKES one, or is 'reps per unit time' an unsupported analogy RDCO should stop asserting in external material?"
Self-correction gate opened by open-follow-up #1 of [[2026-08-03-mac-minimal-feedback-loop-instrumentation]], which stated the selection rule as a rule and traced it to an analogy. Framing correction up front: the MAC framing in that backlog note is dormant (founder, 2026-09-30: "We don't need MaC at all"), so this is not MAC brand work and recommends no MAC positioning. The rule gates two surfaces that are live: Ray's own four-layer prioritization filter (targeting system / instrumentation / tools / feedback loop), which is the filter Ray runs on every new capability question, and Sanity Check, which must not assert an unsupported rule externally.
What we already know (from the vault)
- The exact sentence under audit. [[2026-08-03-mac-minimal-feedback-loop-instrumentation]] §Synthesis: "This also produces the selection rule for which fuzzy decision to instrument first, and it is the opposite of the instinct: instrument the decision you make most often, not the biggest one. The Tesla-fleet analog in [[2026-04-24-targeting-system]] says the agentic targeting system beats the implicit one on reps-per-unit-time, not on stakes." It then reasons that a quarterly loop yields four data points a year and so "will not out-learn a senior operator's priors inside the operator's tenure," concluding "High-frequency, low-stakes decisions are where a cheap loop compounds." The chain is: analogy -> rule -> prescription, with no measurement anywhere in it.
- The cited analogy does not say what the rule says, and this is a category error, not a weak citation. [[2026-04-24-targeting-system]] compares two learners on the same decision: "An individual human driver accumulates at most 1-2 million driving miles over a full career. Tesla's fleet logs on the order of a billion miles per month." Driving is already the high-frequency case for both parties. Nothing in the Tesla frame compares a frequent decision against a rare high-stakes one, which is the comparison the selection rule makes. The fleet argument supports "a machine with more reps beats a human with fewer reps at the same task"; it is silent on "instrument task A rather than task B." The two sentences in the concept doc that come closest to warranting the derived rule do not get there either: the fleet is "capped only by how fast the fleet can log reps and how well the evals convert those reps into learning," and "the same bet is available in any field where enough reps can be logged and graded." Both make rep-rate a threshold condition on whether the bet is available at all - enough reps, gradeable - and the second explicitly pairs reps with grading quality. Neither says more reps makes a decision a better instrumentation target than a rarer, costlier one.
- The same concept doc carries a qualifier the rule drops. [[2026-04-24-targeting-system]] calls "in any domain that permits instrumentation" load-bearing, and states "MAC narrows the fuzz; it doesn't eliminate it. The honest positioning is 'we convert as much of the implicit targeting system into the agentic one as your domain allows.'" The derived rule silently upgrades a domain-permission qualifier into a ranking rule.
- RDCO's own best application of the intuition already used the defensible version of it. [[2026-06-25-fde-proof-point-positioning-signal-instrumentation]] picked its instrument on latency inside the bet's horizon ("reply/reflect-back resonance in the 60-90 day window, versus 6-12 months for inbound/pipeline") and explicitly rejected the high-volume candidate (raw X engagement) as a vanity metric. That is a signal-latency-plus-validity argument, not a frequency-beats-stakes argument. The practice is already better than the stated rule.
- The rule has never shipped as an assertion anywhere. Checked 2026-10-13 with
grep -rln -i -E "reps.per.unit.time|most often, not the (biggest|highest)|frequency.beats.stakes|high.frequency, low.stakes"over~/rdco-vault,~/rdco-hq,~/Documents/phdata-projectsand~/.claude/skills. Hits: the parent brief (the assertion); two unrelated uses of "high-frequency, low-stakes" about payment rails and agent write-paths; and one occurrence in~/rdco-hq/public/research/index.html:752, which is this very question rendered inside ansms:ask link, i.e. the open follow-up, not a claim - and HQ is Access-gated, not public. No Sanity Check issue, landing page, X draft or skill asserts the rule. (Reproducibility note: the command returns 7 file hits rather than 4, because~/rdco-hq/src/content/vault/is a mirror of the vault corpus, not an independent surface - count hits per distinct document.) This is a pre-publication catch, not a retraction.
What the web says
Decomposing the claim into the three things it bundles, as the question demands.
(a) Frequent decisions yield statistically detectable signal faster - FORMAL, effectively arithmetic.
- Required sample per variant is
n = 16σ²/δ²for 80% power; Kohavi, Deng & Vermeer work a real case needing 41,642 users per variant that was run with about 80, giving "power for detecting a 10% relative change" of 3%, which they call "meaningless" (Airbnb-hosted PDF, "A/B Testing Intuition Busters", accessed 2026-10-13). The earlier KDD-2007 form,n = (4rσ/Δ)², states the square factor directly: relaxing the detectable effect by 4x "drops the number of users needed by a factor of 16" (Kohavi, Henne & Sommerfield 2007, accessed 2026-10-13). - Time-to-detect is that
ndivided by the event rate: "As an experiment runs longer, more visitors are admitted into the variants, so sample sizes increase," and narrowing to the 10% of users who trigger a feature cut required users from 1.6M to 256k so "the experiment could run for half the time" (Kohavi & Longbotham, Encyclopedia entry, accessed 2026-10-13). This sub-claim is a rearrangement of the power formula, not an empirical finding. - One strong measured demonstration of the infeasibility end: across 25 large advertising field experiments, "the median confidence interval on return on investment is over 100 percentage points wide," with a coefficient of variation around 10, so "informative advertising experiments can easily require more than 10 million person-weeks" (Lewis & Rao, QJE 2015, accessed 2026-10-13). Note what the binding constraint actually is here: variance relative to effect size, with frequency only one input.
(b) Faster signal produces more learning - MEASURED, but the measured mediator is feedback quality, not frequency.
- Knowledge depreciates, which is the strongest real evidence that infrequent reps decay before they compound. Liberty Ship data from 16 WWII shipyards gives a monthly knowledge-retention parameter where "a value of λ = 0.75 implies that, from a stock of knowledge available at the beginning of a year, only 3.2% would remain one year later," and labour turnover did not explain it; cumulative output "significantly overstates the persistence of learning" (Argote, Beckman & Epple, Management Science 1990, accessed 2026-10-13). Lockheed L-1011 data gives δ = 0.96 monthly, so "61 percent of the firm's stock of experience existing at the beginning of a year survives to the end of the year," and Benkard ties the rate to rarity explicitly: "This high rate of depreciation is in part justified by low aircraft production rates. Some tasks performed by workers are repeated only rarely" (Benkard, NBER WP 7127 / AER 2000, accessed 2026-10-13).
- The scope condition on depreciation matters more to RDCO than the depreciation itself. An architectural/engineering firm with seven years of project data shows λ = 0.9934 monthly with depreciation statistically non-significant; the authors attribute the persistence to a CAD repository of all past designs plus an indexed project database (Boone, Ganeshan & Hicks, Management Science 2008, accessed 2026-10-13). The same paper gives the manufacturing range as 67% to 96% month-to-month retention. Written artifacts suppress the decay that the frequency argument depends on.
- Experimentation adoption is measurably associated with better outcomes, with heavy selection: across ~35,000 startups, A/B testing is "associated with a persistent 11% increase in page views," and adopters "introduce new products at a 9% to 18% higher rate." The specification ladder matters more than the headline: the raw correlation is "280% more page views," which "drops dramatically to 46%" once firm fixed effects are added, with the authors' preferred estimate landing at 10-11% (Koning, Hasan & Chatterji, NBER WP 26278, accessed 2026-10-13). The published Management Science abstract states a different and much larger figure, "performance improves by 30%-100% after a year of use" (Duke scholars listing, accessed 2026-10-13; INFORMS full text paywalled, not retried). Different metrics and specifications; do not blend them. This is observational, not randomized - it is evidence that experimenting firms do better, not that frequency-of-decision is the right instrumentation target.
- The calibration literature's causal variable is explicitly feedback quality and promptness, with frequency riding along. Kahneman & Klein: "Two conditions must be satisfied for skilled intuition to develop: an environment of sufficiently high validity and adequate opportunity to practice the skill," the latter specified as "prolonged practice and feedback that is both rapid and unequivocal" (Kahneman & Klein, American Psychologist 2009, accessed 2026-10-13). Weather forecasters are near-calibrated in field data (gauges that got rain carried mean forecast probability 0.376 vs 0.111 for those that did not; reliability curve near the 45-degree line) (Murphy & Winkler, Atmosphere 1977, accessed 2026-10-13), while clinicians estimate post-positive-radiograph pneumonia at 95% against an evidence range of 46-65% (Morgan et al., JAMA Internal Medicine 2021, accessed 2026-10-13). Training and teaming measurably improved forecast accuracy in the Good Judgment Project (training F(2,1586)=14.29, p<.001; teaming F(2,1586)=60.68, p<.001; Brier means falling from ~0.44 to ~0.23 across conditions) (Mellers et al., Psychological Science 2014, accessed 2026-10-13). Formally, average bandit regret falls as trials accumulate, with regret bounded by 34√(Kn) above and c√((K-1)n) below (Lattimore & Szepesvári, Bandit Algorithms, Thm 9.1 and 13.1, accessed 2026-10-13; the step from those bounds to "average regret → 0" is our arithmetic, not a quoted claim).
- Frequency without validity produces measured anti-learning, which the rule never mentions. Near-misses are frequent and systematically mis-read: managers of near-miss projects are evaluated like managers of successes, so felt risk falls and later choices get riskier (Dillon & Tinsley, Management Science 2008, abstract only, INFORMS paywalled, accessed 2026-10-13). Levinthal & March name the mechanism: "Organizational learning oversamples successes and undersamples failures," and "Experience probably makes nuclear safety engineers over-confident and scientific researchers under-confident" (Levinthal & March 1993, free PDF, accessed 2026-10-13). And fast loops mis-measure by construction: Google found short-horizon experiments capture only about 60% of a learned effect, and that a 50% mobile ad-load reduction was "substantially short-term revenue negative" while long-term neutral (Hohnhold, O'Brien & Tang, KDD 2015, accessed 2026-10-13).
- High velocity also carries a measured false-discovery tax. Across 2,101 Optimizely experiments, "about 73% of experimenters stop the experiment just when a positive effect reaches 90% confidence," and that optional stopping "increases the false discovery rate from 33% to 40%" (Berman, Pekelis, Scott & Van den Bulte, accessed 2026-10-13); separately, across 4,964 effects from 2,766 experiments, "of all effects displaying statistical significance at 5%, about 1 in 5 are truly null" (Berman & Van den Bulte, accessed 2026-10-13). Base rates compound the problem: only about one third of Microsoft ExP ideas "improve the metrics they were designed to improve" (Kohavi, Crook & Longbotham 2009, accessed 2026-10-13).
(c) Instrumenting the frequent decision produces more VALUE than instrumenting the rare high-stakes one - NO measured support found, and the formal result cuts against the "beats" framing.
- Searched and not found - a negative result, not a proof of absence. Across four search framings (including two extended searches) no study was found that compares the payoff of investing measurement effort in high-frequency versus low-frequency decisions. The closest adjacent work is the rare-events subfield, which argues the opposite posture: organizations with meager experience should learn by "treating unique historical incidents as detailed stories rather than single data points," plus near-histories and hypothetical histories (March, Sproull & Tamuz 1991, free 2003 reprint, accessed 2026-10-13) - i.e. rare high-stakes decisions are held to be learnable, by different methods, not written off. The framework paper for the subfield reaches abstract level only (Lampel, Shamsie & Shapira, Organization Science 2009, publisher full text paywalled, accessed 2026-10-13).
- Value-of-information theory makes frequency and stakes multiply, not compete. The expected value of perfect information is zero when the information cannot change the action, and bounded above by EVPI, which scales with the magnitude of the consequence (Gross, Illinois ECE307 Lecture 15; Apostolakis, MIT ESD.72; VPI non-negativity in Russell & Norvig / Berkeley CS188, all accessed 2026-10-13). The frequency term appears explicitly as a multiplier in the health-technology formulation:
Population EVPI = EVPI · Σ_y Q_y(1+α)^-yforQ_ydecisions per year over an H-year horizon (Ades, Lu & Claxton, Medical Decision Making 2004, accessed 2026-10-13). Under that formalism a decision made once a quarter with 1,000x the per-instance stakes dominates a nightly decision; "most frequent" is one of three factors, alongside per-instance stakes and the probability the information changes the action. Howard's original 1966 "Information Value Theory" is paywalled at IEEE and was not retried; the formal statement above is taken from public textbook and lecture sources. - There is a formal result making "many cheap reps" conditional on the tail shape of the idea-quality distribution - and this brief does not establish which branch applies. Azevedo, Deng, Montiel Olea, Rao & Weyl find that when the distribution of innovation quality has moderately thin tails, running a few high-powered experiments remains optimal, and only under very fat tails does testing many ideas with smaller samples dominate (Journal of Political Economy 128(12), 2020, abstract level only, paywalled, accessed 2026-10-13 - do not cite its parameters). That is a conditional result whose branch this brief does not establish - the web-experimentation regime supplying most of the (a) and (b) evidence above may well be the fat-tailed branch, in which case the paper argues for many cheap reps in that regime. It is enough to show that "many cheap reps always wins" is not a formal truth; it is not itself a counterweight, and it is abstract-only.
Convergences and contradictions
- Convergence: the defensible claim is about latency and validity, not frequency. The power formula says detection time equals required sample over event rate; the depreciation studies say unreinforced knowledge decays monthly; Kahneman & Klein say the feedback must be "rapid and unequivocal"; the Google long-term-effect work says a loop shorter than the effect's horizon measures the wrong thing. Assembling those four into a latency rule - a loop is worth installing when it closes faster than the decision recurs and no faster than the outcome actually settles - is this brief's own synthesis step, not a finding any of the four papers states. Flagging it explicitly, because convicting the frequency rule of being analogy-dressed-as-rule and then installing a latency rule on identical footing would repeat the error. What the four do jointly establish is narrower and still useful: no cited source ranks decisions by frequency, and the one source that specifies the conditions for learning specifies feedback quality and promptness rather than count.
- Contradiction: the vault's cited source does not support the vault's rule. [[2026-04-24-targeting-system]] compares learners on one task; [[2026-08-03-mac-minimal-feedback-loop-instrumentation]] converts that into a ranking over tasks. The derived clause "not the biggest one" has no antecedent anywhere in the analogy, and the formal VOI result says the "biggest one" term belongs in the objective as a multiplier.
- Contradiction: the frequency rule's own failure mode is measured, and it is the failure mode RDCO already named. The parent brief warned about "a machine for producing well-calibrated-feeling error, at speed." The near-miss and myopia literature, the 33%-to-40% FDR result and the 1-in-5-null result supply numbers for exactly that warning. The rule and its own kill-switch were published in the same document; only the rule was stated as a rule.
- A possible rescue specific to RDCO, stated as a hypothesis and not a finding: the vault may be a depreciation suppressor. The one measured counterexample to fast knowledge decay credits an indexed repository of past work - but of reusable designs, in a firm with low turnover. Whether that generalizes to judgment-type knowledge is open (see follow-up #2), so this is an untested extrapolation, not support. If it held, it would weaken the "four data points a year cannot compound" argument for RDCO itself more than for a client with no written record, which is the reverse of how the parent brief deployed it.
Synthesis for RDCO
Verdict: PARTIALLY SUPPORTED, and the part RDCO was asserting is the unsupported part. Decomposed: (a) "frequent decisions yield detectable signal faster" is near-tautological - it is the power formula n = 16σ²/δ² divided by the event rate, and the binding term is variance relative to effect size, not frequency alone. (b) "faster signal produces more learning" has genuine measured support, but the measured mediator is feedback that is prompt, unequivocal and from a high-validity environment; frequency is a proxy for that bundle and an unreliable one, since the near-miss and failure-myopia literature documents frequent events producing confidently wrong learning. (c) "and therefore instrumenting the frequent decision beats instrumenting the rare high-stakes one" has no measured support that this brief could find, and the formal value-of-information result refutes the "beats" framing specifically: it makes frequency and per-instance stakes multiply, so neither one dominates a priori. That is a refutation of "frequency beats stakes," not an argument that stakes beat frequency. So the honest statement is the one the dispatch anticipated: the frequency half is near-tautological and the beats-the-highest-stakes half has no measured support.
What RDCO must stop asserting. Retire "instrument the decision you make most often, not the biggest one" as a rule, and retire the Tesla-fleet citation as its warrant. The citation is a category error: the fleet analogy compares two learners on the same high-frequency task and says nothing about choosing between tasks. Also retire the supporting sub-argument that "A loop that resolves once a quarter delivers four data points a year [...] so instrumenting it is a spend with no payback horizon" - under Population EVPI, four reps on a decision with large per-instance consequences can carry more information value than 365 reps on a trivial one, and the rare-events subfield holds that low-frequency decisions are learnable by other means (rich case reconstruction, near-histories, hypothetical histories) rather than not learnable at all.
What RDCO can assert instead, and it is sharper than what it replaces. Three claims. The first and third are traceable to cited sources; the second is RDCO's own assembly of four sources that do not themselves state it, and should be labeled that way wherever it is used. First, the arithmetic one: detection time equals required sample size divided by event rate, and required sample scales with the inverse square of the effect you want to detect, so a loop on a low-rate or high-variance outcome may never resolve inside your decision horizon. That is Kohavi's formula plus Lewis and Rao's requirement of more than 10 million person-weeks (a sample requirement derived from measured variance, not an experiment anyone ran), and it is a feasibility screen, not a ranking. Second, the latency one: instrument a decision only when the loop closes faster than the decision recurs and no faster than the outcome settles - Google's 60%-of-effect finding is the second half, and it is the half that intuition drops. Third, the priority one, which is the honest replacement for the retired rule: rank candidates by frequency times per-instance stakes times the probability the measurement would change the decision, then apply the feasibility screen. That is Population EVPI with Howard's zero-value-if-it-cannot-change-the-action condition attached, and it keeps stakes in the objective where the formal result puts them.
Which live surface this gates. Not MAC - that brand is dormant and nothing here recommends reviving it. Two surfaces. (1) Ray's four-layer prioritization filter ([[2026-04-30-rdco-thesis-targeting-systems-feedback-loops]] and the feedback_targeting_system_prioritization_filter memory): when Ray reaches for the feedback-loop layer to justify instrumenting something, the justification is now the three claims above, not "it is the most frequent thing we do." The filter's own best prior application already did this correctly on latency grounds ([[2026-06-25-fde-proof-point-positioning-signal-instrumentation]]), so this is a tightening, not a reversal. (2) Sanity Check, which per [[2026-04-24-targeting-system]] is meant to hand readers a diagnostic they can apply. The grep says the rule never shipped, so nothing needs correcting in public - the asset is that the replacement claim is publishable and falsifiable where the original was neither. If a Sanity Check issue uses this, the honest shape is the screen-then-rank version carrying its own limits (the FDR numbers and the near-miss anti-learning result as genuine counter-evidence; the fat-tails result as an unresolved conditional rather than a counterweight), which is a better piece than the confident version would have been.
One calibration note on this brief's own limits. Three key sources are abstract-level only (Lampel et al., Dillon & Tinsley, Azevedo et al.), and Howard 1966 is paywalled. The "no direct comparison exists" finding is a failure to find, not a proof of absence; it rests on four search framings on 2026-10-13 and could be overturned by a targeted economics-of-measurement search. Treat the (c) verdict as "unsupported as far as public sources reach," which is enough to stop asserting it and not enough to assert its negation.
Why this is in the vault
It closes the self-correction gate that [[2026-08-03-mac-minimal-feedback-loop-instrumentation]] opened on its own synthesis, and it replaces one retired sentence in Ray's feedback-loop reasoning with three cited, falsifiable claims - which changes how the four-layer prioritization filter justifies instrumenting anything, and gives Sanity Check a publishable version of a claim that would have been indefensible in its original form.
Open follow-ups
- Expected value of sample information and expected net benefit of sampling already subtract study cost in the health-technology-assessment literature (Claxton, co-author of the Population EVPI paper cited here, is a principal author of that strand). Does that cost-subtracted machinery transfer to instrumentation cost - a recurring sensor plus a review ritual, not a one-off study - and has anyone published a worked ENBS-style calculation for an ongoing metric rather than a trial?
- The one measured counterexample to fast knowledge depreciation credits an indexed repository of past work. Is there measured work on how much an organizational memory artifact suppresses the depreciation rate, and does the effect hold for judgment-type knowledge or only for reusable designs?
- The near-miss result says frequent ambiguous events produce confidently wrong updating. Is there a measured threshold - a validity or signal-to-noise level below which adding a feedback loop makes calibration worse than having none? That would convert RDCO's proxy kill-switch from a caveat into a specified trigger.
- Azevedo et al. make "many small experiments versus few powerful ones" depend on the tail thickness of the idea-quality distribution. What is known about estimating that tail shape for a domain before you have run the experiments, which is when the instrumentation choice has to be made?
Related
- [[2026-08-03-mac-minimal-feedback-loop-instrumentation]]
- [[2026-04-24-targeting-system]]
- [[2026-04-30-rdco-thesis-targeting-systems-feedback-loops]]
- [[2026-06-25-fde-proof-point-positioning-signal-instrumentation]]
- [[operational-definitions]]
- [[2026-04-19-commoncog-much-ado-about-the-ooda-loop]]
- [[2026-06-12-agentic-targeting-conviction-calibrated-confidence]]
- [[2026-04-30-amazon-wbr-metric-trees-business-stack-view-research]]
Sources
Vault
- [[2026-08-03-mac-minimal-feedback-loop-instrumentation]] -
~/rdco-vault/06-reference/research/2026-08-03-mac-minimal-feedback-loop-instrumentation.md(the audited sentence, in §Synthesis; its own open-follow-up #1 is this question) - [[2026-04-24-targeting-system]] -
~/rdco-vault/06-reference/concepts/2026-04-24-targeting-system.md(Tesla-fleet analogy as literally written; the "domain that permits instrumentation" qualifier; Sanity Check as public targeting system) - [[2026-04-30-rdco-thesis-targeting-systems-feedback-loops]] -
~/rdco-vault/06-reference/2026-04-30-rdco-thesis-targeting-systems-feedback-loops.md(four-layer prioritization filter, the live surface this gates) - [[2026-06-25-fde-proof-point-positioning-signal-instrumentation]] -
~/rdco-vault/06-reference/research/2026-06-25-fde-proof-point-positioning-signal-instrumentation.md(prior application that used latency, not frequency, and rejected the high-volume candidate) - [[operational-definitions]] -
~/rdco-vault/06-reference/concepts/operational-definitions.md(criterion + test + decision rule; change-log requirement behind the proxy kill-switch)
Web (all accessed 2026-10-13)
- Kohavi, Deng & Vermeer, "A/B Testing Intuition Busters" - https://airbnb.tech/wp-content/uploads/sites/19/2025/10/ABTestingIntuitionBusters.pdf (n = 16σ²/δ²; 41,642-vs-80 underpowered case; 3% power)
- Kohavi, Henne & Sommerfield, KDD 2007, "Practical Guide to Controlled Experiments on the Web" - https://cdn.elezea.com/pdf/GuideControlledExperiments.pdf (n = (4rσ/Δ)²; 4x effect -> 16x sample; triggering 1.6M -> 256k)
- Kohavi & Longbotham, "Online Controlled Experiments and A/B Testing" (encyclopedia entry) - https://netman.aiops.org/~peidan/ANM2017/14.QED/ReadingList/2015Encyclopedia_Online%20Controlled%20Experiments.pdf (sample size as a function of run time; one-third success rate)
- Lewis & Rao, "The Unfavorable Economics of Measuring the Returns to Advertising," QJE 2015 - https://gwern.net/doc/economics/advertising/2015-lewis.pdf (25 experiments; median ROI CI >100pp; CV ~10; >10M person-weeks)
- Argote, Beckman & Epple, Management Science 1990 - https://strategy.sjsu.edu/www.stable/pdf/Argote,%20L,%20S%20Beckman%20and%20D%20Epple,%201990,%20Management%20Science.%2036(2)%20pp%20140-154.pdf (Liberty Ships; λ = 0.75 -> 3.2% of knowledge surviving a year; turnover not the cause)
- Benkard, "Learning and Forgetting: The Dynamics of Aircraft Production," NBER WP 7127 / AER 2000 - https://www.nber.org/papers/w7127.pdf (δ = 0.96 -> 61% annual survival; depreciation attributed to rarely repeated tasks)
- Boone, Ganeshan & Hicks, Management Science 2008 - https://ram-ganeshan.squarespace.com/s/learning-and-knowledge-depreciation-in-professional-services.pdf (λ = 0.9934 with a design repository; 67-96% manufacturing range)
- Koning, Hasan & Chatterji, NBER WP 26278 - https://www.nber.org/system/files/working_papers/w26278.pdf (11% page views as the authors' headline/preferred estimate; 9-18% higher new-product rate; specification ladder 280% raw -> 46% with firm fixed effects -> 10-11% preferred)
- Koning et al., published Management Science abstract - https://scholars.duke.edu/publication/1409782 ("30%-100% after a year"; different specification, do not blend; INFORMS full text paywalled, not retried)
- Kahneman & Klein, "Conditions for Intuitive Expertise," American Psychologist 2009 - https://dl1.cuni.cz/mod/resource/view.php?id=153128 (high validity plus "prolonged practice and feedback that is both rapid and unequivocal")
- Murphy & Winkler, Atmosphere 1977 - https://cmosarchives.ca/Atmosphere/a1502.pdf (forecaster reliability; 0.376 vs 0.111). Note: the canonical Murphy & Winkler 1984 JASA calibration paper is paywalled and was not retried.
- Morgan et al., JAMA Internal Medicine 2021 - https://pubmed.ncbi.nlm.nih.gov/33818595/ (clinician miscalibration: 95% estimated vs 46-65% evidence range)
- Mellers et al., Psychological Science 2014 - https://learnmoore.org/papers/Mellers%20et%20al%202014.pdf (GJP training and teaming F-tests; Brier means ~0.44 -> ~0.23)
- Lattimore & Szepesvári, Bandit Algorithms (2018 draft) - https://ece.iisc.ac.in/~aditya/E1245_Online_Prediction_Learning_F2018/lattimore-szepesvari18bandit-algorithms.pdf (Thm 9.1 upper bound 34√(Kn); Thm 13.1 lower bound c√((K-1)n))
- Dillon & Tinsley, Management Science 2008 - https://ideas.repec.org/a/inm/ormnsc/v54y2008i8p1425-1440.html (near-misses evaluated as successes; ABSTRACT ONLY, INFORMS paywalled)
- Levinthal & March, "The Myopia of Learning," SMJ 1993 - https://arken.nmbu.no/~sigury/AOS_234/AOS234%20files/March_Levinthal_93_The%20myopia%20of%20learning.pdf (failure myopia; "experience rarely generates a rare event")
- March, Sproull & Tamuz, "Learning from Samples of One or Fewer," 2003 reprint of Organization Science 1991 - https://europepmc.org/api/getPdf?pmcid=PMC1758026 (rich histories, near-histories, hypothetical histories; authors' own caveat that these are not guaranteed to improve performance)
- Lampel, Shamsie & Shapira, Organization Science 2009 - https://ideas.repec.org/a/inm/ororsc/v20y2009i5p835-845.html (rare-events learning framework; ABSTRACT ONLY, paywalled)
- Hohnhold, O'Brien & Tang, KDD 2015 - https://research.google.com/pubs/archive/43887.pdf (short-horizon tests capture ~60% of the learned effect; 50% ad-load cut short-term negative, long-term neutral)
- Berman, Pekelis, Scott & Van den Bulte - https://gwern.net/doc/statistics/decision/2018-berman.pdf (2,101 experiments; 73% stop at 90% confidence; FDR 33% -> 40%)
- Berman & Van den Bulte - https://ron-berman.com/papers/fdr.pdf (4,964 effects; ~1 in 5 significant results truly null)
- Kohavi, Crook & Longbotham, 2009 - http://ai.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf (about one third of Microsoft ExP ideas improve their target metric)
- Ades, Lu & Claxton, Medical Decision Making 2004 - https://gwern.net/doc/statistics/decision/2004-ades.pdf (Population EVPI = EVPI · Σ Q_y(1+α)^-y; frequency as a multiplier on per-decision value)
- Gross, Illinois ECE307 Lecture 15 - https://courses.grainger.illinois.edu/ece307/sp2018/handouts/files/Lecture%2015.pdf (EVPI definition; zero value when the action does not change)
- Apostolakis, MIT ESD.72 - https://ocw.iti.hr/courses/engineering-systems-division/esd-72-engineering-risk-benefit-analysis-spring-2007/lecture-notes/da2.pdf (EVPI as an upper bound on willingness to pay)
- Russell & Norvig via Berkeley CS188 textbook - https://inst.eecs.berkeley.edu/~cs188/textbook/vpis/vpi.html (VPI non-negativity). Howard's 1966 "Information Value Theory" itself is paywalled at IEEE and was not retried.
- Azevedo, Deng, Montiel Olea, Rao & Weyl, "A/B Testing with Fat Tails," JPE 2020 - https://www.journals.uchicago.edu/doi/abs/10.1086/710607 (few high-powered experiments optimal under thin tails; ABSTRACT ONLY, paywalled, parameters not cited)