Can instrumentation manufacture conviction in fuzzy domains, or does conviction stay human?
The question
Can an agentic targeting system (evals, benchmarks, test harnesses) provide the conviction for human decision-makers to push ahead in un-instrumented situations — or is conviction the one thing that stays with the implicit (taste/experience) targeting system? Does any published work show AI outputs leading humans to commit in fuzzy domains at a rate that matches or beats senior-operator conviction? (Surfaced 2026-04-24 in the RDCO Targeting System synthesis; this is where MAC's positioning earns its keep.)
What we already know (from the vault)
- The [[2026-04-24-targeting-system]] concept doc names this as a load-bearing open question, explicitly unsolved: the agentic targeting system works when outcomes are well-defined, but the residual fuzz — calls that can't be reduced to a spec in advance — is "where conviction still lives," in the founder/senior-operator earned-track-record layer. The honest positioning is already drafted: MAC "narrows the fuzz; it doesn't eliminate it."
- The founder's own framing splits decision-making into implicit (taste/experience, validated by track record, no eval required) vs agentic (evals/harness, requires measurable outcomes). They are "phase-matched to different domains," not substitutes. The Tesla-FSD analog holds only "in any domain that permits instrumentation" — that qualifier is the whole ballgame.
- The harness-thesis cluster ([[2026-05-12-garry-tan-ai-agent-complexity-ratchet-90-test-coverage]]) argues evals + tests + docs ratchet quality upward in instrumentable domains (code, data pipelines). It is silent on un-instrumented conviction — it is evidence for the agentic side of the split, not the implicit side.
- The vault treats uncertainty as a ground state, not a solvable problem ([[2026-04-19-commoncog-five-sources-of-uncertainty]]); senior judgment under scarce ground truth is a feature of business, not a temporary gap.
What the web says
- LLMs do not match senior-operator conviction in fuzzy domains — quantitatively. On real-world prediction-market forecasting, frontier models trail expert forecasters badly: o3 Brier 0.135, GPT-4.1 0.154, vs expert median 0.023; o3 only marginally beats the human crowd baseline (0.149) and "significantly underperforms a group of experts." Authors conclude LLM forecasts "remain unsuitable for high-stakes decisions requiring human expert judgment." (arxiv 2507.04562)
- Models are systematically overconfident, exactly where it matters. They predict near-certainty for events with observed frequency ~50%; expert superforecasters hit ECE ≈0.03–0.05. Extended reasoning produces "more text but not necessarily better uncertainty quantification" — models optimize for persuasive reasoning, not calibrated forecasting. Practitioner guidance: when a model says 90%+, expect 20–30% error rates. (arxiv 2503.15850, arxiv 2512.16030)
- Showing AI confidence DOES move human commitment — but through a contaminating mechanism, not earned conviction. The CHI 2025 "As Confidence Aligns" study finds human self-confidence "aligns with AI confidence and such alignment can persist even after AI ceases to be involved." Humans unconsciously adopt the model's confidence level without a change in their actual decision capability — so the AI imports its own miscalibration into the human. (arxiv 2501.12868)
- Verbalized confidence is real but doesn't convert to good decisions. Models can verbalize calibrated-ish confidence (prompt- and model-agnostic, cheap), yet "fail to translate this signal into decision-making" and "lack the strategic agency to convert uncertainty signals into risk-sensitive decisions." Calibrated confidence is necessary, not sufficient, for trustworthy AI. (arxiv 2412.14737)
- Appropriate-reliance research says expertise is the moderator. Higher human expertise correlates with higher self-confidence and lower trust in AI advice (a more critical stance); reliance is governed by the operator's own self-confidence, not just the AI's confidence display. Showing AI confidence has limited effect precisely because senior operators weight their own priors. (tandfonline 2025)
- Real-time correctness feedback is the one lever that breaks the bad alignment — i.e., instrumentation helps only when outcomes become observable (an instrumented domain), which is the same boundary the vault already drew. (arxiv 2501.12868)
Convergences and contradictions
- Strong convergence: The published evidence ratifies the vault's unsolved-question framing. In un-instrumented/fuzzy domains, AI confidence is (a) miscalibrated and overconfident, (b) below expert human calibration, and (c) capable of moving human commitment only by importing its own error into the human — not by manufacturing well-grounded conviction. No published work shows AI outputs driving humans to commit in fuzzy domains at a rate matching senior-operator conviction; where it moves them, it degrades calibration.
- Sharpening, not contradiction: The web adds a mechanism the vault didn't have — "confidence alignment." This is the dangerous case for RDCO to name: an agentic system can produce false conviction (humans defer to a confident-sounding model), which is worse than no conviction. The vault's "narrows the fuzz" framing should now explicitly warn against manufactured-but-miscalibrated conviction.
- Boundary agreement: Both sides agree the fix is observable outcomes (real-time feedback / instrumentation). Conviction becomes transferable to AI exactly when and only when the domain becomes instrumentable — which by definition means it has left the fuzzy category.
Synthesis for RDCO
The published evidence comes down on the side the founder already suspected: conviction in genuinely un-instrumented situations stays human, and the targeting-system frame is correct to treat that residual as the durable seat of senior judgment. As of mid-2026 there is no work showing AI matching or beating senior-operator conviction in fuzzy domains. Frontier models trail expert forecasters by roughly 6x on Brier score, are overconfident exactly where stakes are highest, and — most importantly — can verbalize uncertainty but cannot convert it into risk-sensitive commitment. The mechanism by which AI confidence does move humans ("confidence alignment") is a contamination effect, not a conviction-transfer: it changes how sure the human feels without changing whether they should be. That is the opposite of what a trustworthy targeting system would do.
This tightens, rather than weakens, MAC / agent-deployer positioning — provided RDCO sells the honest version. The defensible, evidence-backed pitch is the one the vault already drafted: "we convert as much of the implicit targeting system into the agentic one as your domain allows." MAC's job is to expand the instrumentable frontier — turn taste into acceptance criteria, "good enough" into a gradeable spec, anomaly-escalation instinct into eval-backed thresholds — and the published evidence shows this is real value precisely because real-time observable feedback is the one lever that fixes miscalibrated reliance. MAC is in the business of manufacturing the feedback loop that makes conviction transferable. What it must never claim is that it manufactures conviction in the un-instrumented residual; the calibration literature would falsify that on contact, and a confident-but-miscalibrated agent is a liability, not a feature.
The strategic refinement: RDCO should treat "the un-instrumented residual" not as a permanent fixed quantity but as a moving frontier MAC pushes outward, while explicitly conceding the frontier never reaches zero. Two assets fall out of this. First, a credibility moat — RDCO can say, with citations, exactly why a vendor promising "AI conviction in fuzzy domains" is overselling (overconfidence + confidence-alignment contamination), which is a sharp Sanity Check re-frame and a trust-building move in sales. Second, a product wedge — Client Reporting and MAC should be sold as instrumentation services that earn the right to transfer conviction, with the explicit boundary that the final "proceed under genuine ambiguity" call stays with the human until outcomes become observable. That boundary is not a weakness to hide; per the appropriate-reliance research, senior operators trust vendors who respect their judgment-seat more, so naming the boundary is itself a positioning advantage.
Open follow-ups
- Does the LLM-vs-superforecaster Brier gap narrow in domains where the model has tool-access to live ground truth (search, fleet data) — i.e., is the "instrumentable frontier" expanding fast enough to re-price the residual? (candidate /curiosity)
- Is there published work on agent ensembles / debate / market mechanisms recovering expert-level calibration where single models fail — could RDCO instrument conviction via a multi-agent targeting system rather than a single eval?
- "Confidence alignment" as a sales-ethics risk: should RDCO's deliverables deliberately suppress unwarranted AI confidence displays to protect client calibration? Is "calibration-protective UX" a productizable MAC differentiator?
- Quantify the Tesla-FSD analog for RDCO's own vault-as-fleet bet: does processed-rep volume measurably improve the founder's decision calibration, and is that even instrumentable?
- What is the minimal instrumentation that flips a "fuzzy" client decision into one where real-time feedback breaks the bad alignment — i.e., what is MAC's cheapest possible feedback-loop install?
Related
- [[2026-04-24-targeting-system]]
- [[2026-05-12-garry-tan-ai-agent-complexity-ratchet-90-test-coverage]]
- [[2026-04-19-commoncog-five-sources-of-uncertainty]]
- [[2026-04-03-superforecasting-framework]]
- [[2026-04-30-quality-gate-as-brain-org-boundaries-agentic-companies]]
- [[2026-04-08-better-harness-evals-hill-climbing]]
- [[2026-04-11-garry-tan-thin-harness-fat-skills]]
Sources
- Vault: [[2026-04-24-targeting-system]] —
~/rdco-vault/06-reference/concepts/2026-04-24-targeting-system.md(the conviction-in-uncertainty gap, implicit vs agentic split, MAC positioning) - Vault: [[2026-05-12-garry-tan-ai-agent-complexity-ratchet-90-test-coverage]] —
~/rdco-vault/06-reference/2026-05-12-garry-tan-ai-agent-complexity-ratchet-90-test-coverage.md(harness/eval ratchet in instrumentable domains) - Vault: [[2026-04-19-commoncog-five-sources-of-uncertainty]] —
~/rdco-vault/06-reference/2026-04-19-commoncog-five-sources-of-uncertainty.md(uncertainty as ground state) - Vault: [[2026-04-03-superforecasting-framework]] —
~/rdco-vault/06-reference/2026-04-03-superforecasting-framework.md(Tetlock calibration baseline) - Web: Evaluating LLMs on Real-World Forecasting Against Human Superforecasters — https://arxiv.org/html/2507.04562v3
- Web: Uncertainty Quantification and Confidence Calibration in LLMs: A Survey — https://arxiv.org/html/2503.15850v1
- Web: Do LLMs Know What They Don't Know? (Epistemic Calibration via Prediction Markets) — https://arxiv.org/html/2512.16030
- Web: As Confidence Aligns: Effect of AI Confidence on Human Self-confidence (CHI 2025) — https://arxiv.org/abs/2501.12868
- Web: On Verbalized Confidence Scores for LLMs — https://arxiv.org/html/2412.14737v2
- Web: Investigating Appropriate Reliance on AI-Based Decision Support (expertise/trust/self-confidence) — https://www.tandfonline.com/doi/full/10.1080/12460125.2025.2593251