06-reference/research

frequent vs high stakes decision instrumentation evidence

2026-10-13·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
self-correctioninstrumentationfeedback-looptargeting-systemvalue-of-information

The frequency half is arithmetic, the "beats the highest-stakes one" half has no measured support, and the formal result makes frequency and stakes multiply rather than compete

The question

"Is there measured evidence that instrumenting the most FREQUENT decision beats the highest-STAKES one, or is 'reps per unit time' an unsupported analogy RDCO should stop asserting in external material?"

Self-correction gate opened by open-follow-up #1 of [[2026-08-03-mac-minimal-feedback-loop-instrumentation]], which stated the selection rule as a rule and traced it to an analogy. Framing correction up front: the MAC framing in that backlog note is dormant (founder, 2026-09-30: "We don't need MaC at all"), so this is not MAC brand work and recommends no MAC positioning. The rule gates two surfaces that are live: Ray's own four-layer prioritization filter (targeting system / instrumentation / tools / feedback loop), which is the filter Ray runs on every new capability question, and Sanity Check, which must not assert an unsupported rule externally.

What we already know (from the vault)

What the web says

Decomposing the claim into the three things it bundles, as the question demands.

(a) Frequent decisions yield statistically detectable signal faster - FORMAL, effectively arithmetic.

(b) Faster signal produces more learning - MEASURED, but the measured mediator is feedback quality, not frequency.

(c) Instrumenting the frequent decision produces more VALUE than instrumenting the rare high-stakes one - NO measured support found, and the formal result cuts against the "beats" framing.

Convergences and contradictions

Synthesis for RDCO

Verdict: PARTIALLY SUPPORTED, and the part RDCO was asserting is the unsupported part. Decomposed: (a) "frequent decisions yield detectable signal faster" is near-tautological - it is the power formula n = 16σ²/δ² divided by the event rate, and the binding term is variance relative to effect size, not frequency alone. (b) "faster signal produces more learning" has genuine measured support, but the measured mediator is feedback that is prompt, unequivocal and from a high-validity environment; frequency is a proxy for that bundle and an unreliable one, since the near-miss and failure-myopia literature documents frequent events producing confidently wrong learning. (c) "and therefore instrumenting the frequent decision beats instrumenting the rare high-stakes one" has no measured support that this brief could find, and the formal value-of-information result refutes the "beats" framing specifically: it makes frequency and per-instance stakes multiply, so neither one dominates a priori. That is a refutation of "frequency beats stakes," not an argument that stakes beat frequency. So the honest statement is the one the dispatch anticipated: the frequency half is near-tautological and the beats-the-highest-stakes half has no measured support.

What RDCO must stop asserting. Retire "instrument the decision you make most often, not the biggest one" as a rule, and retire the Tesla-fleet citation as its warrant. The citation is a category error: the fleet analogy compares two learners on the same high-frequency task and says nothing about choosing between tasks. Also retire the supporting sub-argument that "A loop that resolves once a quarter delivers four data points a year [...] so instrumenting it is a spend with no payback horizon" - under Population EVPI, four reps on a decision with large per-instance consequences can carry more information value than 365 reps on a trivial one, and the rare-events subfield holds that low-frequency decisions are learnable by other means (rich case reconstruction, near-histories, hypothetical histories) rather than not learnable at all.

What RDCO can assert instead, and it is sharper than what it replaces. Three claims. The first and third are traceable to cited sources; the second is RDCO's own assembly of four sources that do not themselves state it, and should be labeled that way wherever it is used. First, the arithmetic one: detection time equals required sample size divided by event rate, and required sample scales with the inverse square of the effect you want to detect, so a loop on a low-rate or high-variance outcome may never resolve inside your decision horizon. That is Kohavi's formula plus Lewis and Rao's requirement of more than 10 million person-weeks (a sample requirement derived from measured variance, not an experiment anyone ran), and it is a feasibility screen, not a ranking. Second, the latency one: instrument a decision only when the loop closes faster than the decision recurs and no faster than the outcome settles - Google's 60%-of-effect finding is the second half, and it is the half that intuition drops. Third, the priority one, which is the honest replacement for the retired rule: rank candidates by frequency times per-instance stakes times the probability the measurement would change the decision, then apply the feasibility screen. That is Population EVPI with Howard's zero-value-if-it-cannot-change-the-action condition attached, and it keeps stakes in the objective where the formal result puts them.

Which live surface this gates. Not MAC - that brand is dormant and nothing here recommends reviving it. Two surfaces. (1) Ray's four-layer prioritization filter ([[2026-04-30-rdco-thesis-targeting-systems-feedback-loops]] and the feedback_targeting_system_prioritization_filter memory): when Ray reaches for the feedback-loop layer to justify instrumenting something, the justification is now the three claims above, not "it is the most frequent thing we do." The filter's own best prior application already did this correctly on latency grounds ([[2026-06-25-fde-proof-point-positioning-signal-instrumentation]]), so this is a tightening, not a reversal. (2) Sanity Check, which per [[2026-04-24-targeting-system]] is meant to hand readers a diagnostic they can apply. The grep says the rule never shipped, so nothing needs correcting in public - the asset is that the replacement claim is publishable and falsifiable where the original was neither. If a Sanity Check issue uses this, the honest shape is the screen-then-rank version carrying its own limits (the FDR numbers and the near-miss anti-learning result as genuine counter-evidence; the fat-tails result as an unresolved conditional rather than a counterweight), which is a better piece than the confident version would have been.

One calibration note on this brief's own limits. Three key sources are abstract-level only (Lampel et al., Dillon & Tinsley, Azevedo et al.), and Howard 1966 is paywalled. The "no direct comparison exists" finding is a failure to find, not a proof of absence; it rests on four search framings on 2026-10-13 and could be overturned by a targeted economics-of-measurement search. Treat the (c) verdict as "unsupported as far as public sources reach," which is enough to stop asserting it and not enough to assert its negation.

Why this is in the vault

It closes the self-correction gate that [[2026-08-03-mac-minimal-feedback-loop-instrumentation]] opened on its own synthesis, and it replaces one retired sentence in Ray's feedback-loop reasoning with three cited, falsifiable claims - which changes how the four-layer prioritization filter justifies instrumenting anything, and gives Sanity Check a publishable version of a claim that would have been indefensible in its original form.

Open follow-ups

Related

Sources

Vault

Web (all accessed 2026-10-13)