06-reference/research

cortex sense accuracy lift ablation

2026-11-05·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
cortex-sensetext-to-sqlorganizational-intelligenceablationevidence-discipline

Nobody Has Ablated Cortex Sense. Two 2026 Papers Ablate the Same Variable on Public Benchmarks, and They Bound the Organizational-Knowledge Share at 13 to 20 Execution-Accuracy Points, Not 60

The question

"What share of Cortex Sense's reported 23%→83% accuracy lift comes from organizational knowledge vs. schema-inferable knowledge a model could learn unaided — has anyone published an ablation?"

Context: this was open follow-up #2 in [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]] and the named "most useful open question" in [[2026-09-12-pupius-four-axis-snowflake-data-ai-strategy]]. It is load-bearing for the Organizational Intelligence (OI) wedge, whose premise is that organizational knowledge not inferable from schema is the durable moat.

Headline: no ablation of Cortex Sense exists and Snowflake's comparison design cannot produce one. But two papers published in 2026 run exactly this ablation on public benchmarks, holding model and pipeline fixed, and both land the organizational-knowledge contribution an order of magnitude below Snowflake's headline gap.

What we already know (from the vault)

What the web says

Convergences and contradictions

Synthesis for RDCO

The answer to the question as asked is that the split has still not been published for Cortex Sense, and it cannot be: Snowflake's experiment varies the harness while holding knowledge constant, so no amount of re-reading the blog recovers a knowledge-versus-schema ratio. What changed in the last two months is that the ablation now exists on public benchmarks. BIRD-History and EntSQL both hold the model and pipeline fixed and toggle only access to organizational knowledge, and that is the design Snowflake declined to run. The usable result is a bracket rather than a point estimate: with model and harness constant, organizational knowledge is worth roughly 13 execution-accuracy points on a benchmark where schema carries most questions, and roughly 15 points (6.8 to 21.4) on a benchmark engineered so schema carries almost none. Both are an order of magnitude below the 60-point gap the Snowflake line implies. The honest sentence is that the 23-to-83 gap is mostly harness and retrieval, with an organizational-knowledge core in the low double digits.

That is a smaller number than the OI pitch has been implicitly leaning on, and a much more durable one. The two halves of the gap behave differently over time, and this is the finding worth carrying forward. The schema-inferable half sits on a steep capability escalator: dbt Labs measured raw text-to-SQL going from 32.7% to 64.5% in three years with no semantic layer at all. Anyone whose wedge is "the model cannot read your schema well enough" is standing on ground that moves about ten points a year. The organizational-knowledge half is flat with respect to model strength: eight frontier models cluster inside 2.1 points on EntSQL's question-plus-schema setting, Opus 4.6 does not beat Sonnet 4.6 there, and models finetuned to 67-69% on BIRD dev fall to 19-29% on the same questions once organizational context is what is missing. A 13-point effect that does not decay is strictly better positioning than a 60-point effect that is mostly plumbing a vendor will ship for free.

BIRD-History hands us a taxonomy the Intelligence Maturity Assessment should adopt verbatim, and it independently validates the component the vault already treated as first-class. Its five types are calculation, condition, relation, dimension and output, and the paper measures that relation (join paths) and calculation (aggregation formulas) are the least recoverable from the question, both under 30% retrieval precision, with the explicit finding that join paths "are often non-obvious from schema alone." [[concepts/semantic-layer|semantic-layer]] already holds approved join paths as a component distinct from metric definitions, and [[2026-02-04-practical-data-modeling-dead-again-part2]] already argued agents lack context about arcane business operations. There is now a measured, third-party, non-Snowflake basis for saying which two kinds of organizational knowledge are the irreducible ones. The Confidently-Wrong Probe proposed in [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]] should score per type against these five labels, because a client report that says "your estate is missing relation and calculation knowledge, and here are the wrong numbers it produced" is a diagnosis, while "your accuracy is 61%" is a statistic.

Operationally, three things follow. First, retire the 23-to-83 line from OI materials entirely rather than re-framing it; the replacement claim is "published ablations that hold the model fixed put governed organizational context at 13 to 20 accuracy points, and that part does not shrink as models improve," which survives a client checking the citation because the citations are independent of Snowflake. Second, the per-type framing is a Sanity Check piece with an original re-frame, the decomposition itself, rather than a vendor number re-stated. Third, BIRD-History publishes its annotations at github.com/zjuidg/BIRD-History, which means we can run the None-versus-Gold contrast ourselves per knowledge type and produce the per-type generation deltas the paper reports only as retrieval F1. That is a cheap, publishable, first-party measurement in a gap the literature has left open, and it is the same instrument-not-statistic move the parent brief recommended.

Why this is in the vault

This closes open follow-up #2 of [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]] and the explicitly-named unknown in [[2026-09-12-pupius-four-axis-snowflake-data-ai-strategy]], and it replaces the retired 23-to-83 line with a defensible magnitude from two non-Snowflake 2026 ablations. It changes what a phData DSA may assert about semantic-layer value (13 to 20 points, durable, not 60) and it supplies the five-type scoring taxonomy for the Confidently-Wrong Probe inside the Intelligence Maturity Assessment.

Open follow-ups

Related

Sources

Vault:

Web (fetched or searched 2026-11-05):