Nobody Has Ablated Cortex Sense. Two 2026 Papers Ablate the Same Variable on Public Benchmarks, and They Bound the Organizational-Knowledge Share at 13 to 20 Execution-Accuracy Points, Not 60
The question
"What share of Cortex Sense's reported 23%→83% accuracy lift comes from organizational knowledge vs. schema-inferable knowledge a model could learn unaided — has anyone published an ablation?"
Context: this was open follow-up #2 in [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]] and the named "most useful open question" in [[2026-09-12-pupius-four-axis-snowflake-data-ai-strategy]]. It is load-bearing for the Organizational Intelligence (OI) wedge, whose premise is that organizational knowledge not inferable from schema is the durable moat.
Headline: no ablation of Cortex Sense exists and Snowflake's comparison design cannot produce one. But two papers published in 2026 run exactly this ablation on public benchmarks, holding model and pipeline fixed, and both land the organizational-knowledge contribution an order of magnitude below Snowflake's headline gap.
What we already know (from the vault)
- The headline pair is a vendor self-report with a moving baseline, and the vault has already retired it. [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]] traced it to Snowflake's own post, which reports 24.1% to 86.3% and compares "a frontier coding agent with direct access to SQL execution via model context protocol (MCP), vanilla CoCo ... and CoCo grounded by Cortex Sense" on "product analytics questions on Snowflake's internal data." Across Snowflake surfaces the same baseline appears as 21%, 23%, 24.1% and about 25%. [[2026-09-23-correctness-criticality-cannibalization-threshold]] states the figure is "a ceiling on Snowflake's own data, not a floor on the client's."
- Snowflake's comparison varies the harness, not the knowledge. Same brief: the baseline is an external agent reaching in through a connector with no retrieval layer, against a grounded agent, on a constant data estate. Nothing in that design isolates knowledge type, which is the structural reason the requested split cannot be recovered from the published numbers.
- The vault already named the split as unpublished. [[2026-09-12-pupius-four-axis-snowflake-data-ai-strategy]]: "part of the 23%-to-83% gap is permissions, lineage and business definitions ... The exposed share is the portion a better model could infer from schema and query history. Neither Snowflake nor anyone else publishes that split."
- The vault's one contradicting benchmark also contains a model-capability time series nobody has used yet. [[2026-04-07-dbt-semantic-layer-vs-text-to-sql-benchmark]] measured semantic layer at 98.2-100% against raw text-to-SQL at 84.1-90% on an 11-question ACME Insurance set, a 10-15 point gap. It also reports raw text-to-SQL "doubling from 32.7% in 2023 to 64.5% in 2026" and that even three dbt models improved both approaches.
- The wedge was banked on the governed contract plus eval, not on the gap size. [[2026-09-23-semantic-studio-self-serve-authoring-erosion]] found the half actually being commoditized is eval, and [[concepts/semantic-layer|semantic-layer]] holds approved join paths as a first-class component distinct from metric definitions.
What the web says
- No ablation of Cortex Sense, Cortex Analyst or any CoWork plugin was found, and none is methodologically possible from the published artifacts. Snowflake's most transparent accuracy post claims "90%+ SQL accuracy" on a 150-question internal suite but scopes itself to "a single view with pre-joined data" and publishes no with-model versus without-model delta (Snowflake engineering blog, via [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]]).
- BIRD-History (arXiv 2608.29345v1, Zhou, Shi, Yang, Weng and Wu) is the cleanest published version of the requested ablation. Knowledge is extracted from an organization's historical SQL queries, the same input class Cortex Sense consumes. Its Table 13 holds pipeline and model fixed (OpenSearch-SQL on Qwen3) and varies only access to that knowledge: no retriever 54.27% execution accuracy, its retriever 60.95%, oracle gold annotations 67.34%. So organizational knowledge is worth +13.07 points at a perfect-retrieval ceiling and +6.68 points as actually retrievable today, against a 54.27% schema-only floor. My computation from those figures: the organizational-knowledge share of the oracle score is 19.4%, the schema-plus-model share is 80.6%.
- BIRD-History also publishes a five-type knowledge taxonomy, and the two types least recoverable without history are exactly the ones the OI wedge claims. Types are calculation (SELECT and ORDER BY formulas), condition (WHERE and HAVING predicates), relation (FROM-JOIN paths), dimension (GROUP BY granularity) and output (ORDER BY, LIMIT, DISTINCT). Retrieval F1 by type: condition 49.64, output 48.06, dimension 43.77, relation 38.73, calculation 35.75. The paper states relation and calculation knowledge show precision "both below 30%" because "natural language questions rarely specify exact JOIN paths or aggregation formulas explicitly," and that join paths "are often non-obvious from schema alone."
- EntSQL (arXiv 2606.03363v2, 6 Jul 2026; Liao of HKUST-GZ with Xu, Chen, Wang, Wang, Zhang, Chen and Yang of Alibaba Group and Xu of the Guangdong Laboratory) runs the same contrast and lands on the opposite side. 1,066 aligned Chinese-English examples over five business domains, grounded in "internal metrics, reporting conventions, and organizational rules." Question-plus-schema base accuracy across eight frontier models is 4.7% to 6.8%; adding long-form domain documents takes the best to 15.9%; expert-curated evidence takes it to 21.4%. The paper: "Adding long-form domain documents improves the best Avg accuracy from 6.8% to 15.9%, while providing expert-curated evidence further increases it to 21.4%."
- EntSQL is the strongest published evidence that a stronger base model does not buy the organizational-knowledge half. Its eight-model base band spans 2.1 points (4.7% to 6.8%) while the evidence delta spans 5.8 to 14.6 points, and Opus 4.6 (5.9% base, 15.5% with evidence) does not beat Sonnet 4.6 (5.9%, 17.6%). Its conclusion: "effective knowledge selection and grounding, rather than simply increasing context length, is a key bottleneck." Expert humans on a 212-example subset score 33.5%, 46.2% and 84.0% across the three settings, so headroom is knowledge-shaped, not model-shaped.
- Capability trained against schema-only benchmarks transfers badly to organization-grounded ones. BIRD-History Table 11: five text-to-SQL-finetuned models score 67.01% to 69.23% on BIRD dev but 19.02% to 28.64% on BIRD-History. Table 12: foundation models score 16.65% (Llama3.3-70B-Instruct) to 33.24% (Gemini-3-Flash) on BIRD-History against 35.07% to 51.56% on BIRD dev.
- Counter-evidence, stated plainly: model choice is not negligible. That same Table 12 spread is 16.6 points wide, which exceeds the 13.07-point oracle-knowledge ceiling in Table 13. The two tables are not directly comparable (different pipelines and models), so this bounds the claim rather than refuting it, but any version of the wedge argument that says model choice does not matter is overclaiming.
- The literature structurally hides the no-knowledge condition, which is why this question stayed open. Two claims reached me only through a search-result summary and I could not confirm either at primary source, so both are flagged unverified: that the original BIRD paper reports GPT-4 at 34.88% without evidence against 54.89% with it (+20.01 points), and that a 2026 paper named DIVER found a greater-than-10-point collapse across model sizes plus "among the 52 methods, only 5 methods report performance without evidence" on the BIRD leaderboard. The one candidate arXiv identifier I fetched (2603.07841) contained neither claim. Separately, Spider 2.0's own paper is reported to reach only 10.98% on tasks requiring external documents, also unverified here (Spider 2.0, ICLR 2025).
Convergences and contradictions
- Convergence on direction, sharp contradiction on magnitude, and the contradiction is informative rather than fatal. BIRD-History puts organizational knowledge at 19.4% of the achievable score; EntSQL puts it near two thirds (6.8 of 21.4 best-case). Neither is wrong. BIRD-History questions are answerable from schema most of the time, so schema carries the load; EntSQL is constructed so private documents are required, so it is an upper bound by design. The organizational-knowledge share is a property of the question set, not of the technology - which is the same conclusion [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]] reached from the opposite direction when it found "un-modeled" is a property of the client's estate.
- Both published ablations contradict Snowflake's implied magnitude. Neither comes close to a 60-point knowledge effect with model and harness held constant. The residual in Snowflake's 24.1-to-86.3 gap is most consistent with retrieval and harness plumbing, an external agent with no context mechanism against a grounded one, which is exactly the design the vault already identified.
- Convergence against the "stronger model absorbs it" hypothesis, with one honest exception. EntSQL's 2.1-point frontier band and BIRD-History's finetuned-model collapse both say capability does not substitute for organizational knowledge. The exception is dbt Labs' own time series: raw text-to-SQL rose from 32.7% in 2023 to 64.5% in 2026 ([[2026-04-07-dbt-semantic-layer-vs-text-to-sql-benchmark]]), roughly 32 points in three years with no semantic layer. That is the schema-inferable half being absorbed in public, on schedule.
Synthesis for RDCO
The answer to the question as asked is that the split has still not been published for Cortex Sense, and it cannot be: Snowflake's experiment varies the harness while holding knowledge constant, so no amount of re-reading the blog recovers a knowledge-versus-schema ratio. What changed in the last two months is that the ablation now exists on public benchmarks. BIRD-History and EntSQL both hold the model and pipeline fixed and toggle only access to organizational knowledge, and that is the design Snowflake declined to run. The usable result is a bracket rather than a point estimate: with model and harness constant, organizational knowledge is worth roughly 13 execution-accuracy points on a benchmark where schema carries most questions, and roughly 15 points (6.8 to 21.4) on a benchmark engineered so schema carries almost none. Both are an order of magnitude below the 60-point gap the Snowflake line implies. The honest sentence is that the 23-to-83 gap is mostly harness and retrieval, with an organizational-knowledge core in the low double digits.
That is a smaller number than the OI pitch has been implicitly leaning on, and a much more durable one. The two halves of the gap behave differently over time, and this is the finding worth carrying forward. The schema-inferable half sits on a steep capability escalator: dbt Labs measured raw text-to-SQL going from 32.7% to 64.5% in three years with no semantic layer at all. Anyone whose wedge is "the model cannot read your schema well enough" is standing on ground that moves about ten points a year. The organizational-knowledge half is flat with respect to model strength: eight frontier models cluster inside 2.1 points on EntSQL's question-plus-schema setting, Opus 4.6 does not beat Sonnet 4.6 there, and models finetuned to 67-69% on BIRD dev fall to 19-29% on the same questions once organizational context is what is missing. A 13-point effect that does not decay is strictly better positioning than a 60-point effect that is mostly plumbing a vendor will ship for free.
BIRD-History hands us a taxonomy the Intelligence Maturity Assessment should adopt verbatim, and it independently validates the component the vault already treated as first-class. Its five types are calculation, condition, relation, dimension and output, and the paper measures that relation (join paths) and calculation (aggregation formulas) are the least recoverable from the question, both under 30% retrieval precision, with the explicit finding that join paths "are often non-obvious from schema alone." [[concepts/semantic-layer|semantic-layer]] already holds approved join paths as a component distinct from metric definitions, and [[2026-02-04-practical-data-modeling-dead-again-part2]] already argued agents lack context about arcane business operations. There is now a measured, third-party, non-Snowflake basis for saying which two kinds of organizational knowledge are the irreducible ones. The Confidently-Wrong Probe proposed in [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]] should score per type against these five labels, because a client report that says "your estate is missing relation and calculation knowledge, and here are the wrong numbers it produced" is a diagnosis, while "your accuracy is 61%" is a statistic.
Operationally, three things follow. First, retire the 23-to-83 line from OI materials entirely rather than re-framing it; the replacement claim is "published ablations that hold the model fixed put governed organizational context at 13 to 20 accuracy points, and that part does not shrink as models improve," which survives a client checking the citation because the citations are independent of Snowflake. Second, the per-type framing is a Sanity Check piece with an original re-frame, the decomposition itself, rather than a vendor number re-stated. Third, BIRD-History publishes its annotations at github.com/zjuidg/BIRD-History, which means we can run the None-versus-Gold contrast ourselves per knowledge type and produce the per-type generation deltas the paper reports only as retrieval F1. That is a cheap, publishable, first-party measurement in a gap the literature has left open, and it is the same instrument-not-statistic move the parent brief recommended.
Why this is in the vault
This closes open follow-up #2 of [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]] and the explicitly-named unknown in [[2026-09-12-pupius-four-axis-snowflake-data-ai-strategy]], and it replaces the retired 23-to-83 line with a defensible magnitude from two non-Snowflake 2026 ablations. It changes what a phData DSA may assert about semantic-layer value (13 to 20 points, durable, not 60) and it supplies the five-type scoring taxonomy for the Confidently-Wrong Probe inside the Intelligence Maturity Assessment.
Open follow-ups
- Run the BIRD-History None-versus-Gold contrast per knowledge type ourselves. The paper reports retrieval precision and recall by type but no execution-accuracy delta by type; the annotations are released at github.com/zjuidg/BIRD-History, so the per-type generation delta is a tractable first-party measurement nobody has published.
- Verify the original BIRD with-evidence versus without-evidence pair (reported as GPT-4 at 34.88% against 54.89%) at primary source, arXiv 2305.03111 or the NeurIPS 2023 paper. This brief carries it as secondary and unverified.
- Confirm or discard the "DIVER" claims: a greater-than-10-point no-evidence collapse across model sizes, and "only 5 of 52" BIRD leaderboard methods reporting a no-evidence number. Both reached this brief only through a search summary and the candidate arXiv identifier did not contain them. The "5 of 52" claim, if real, is the single best one-line explanation of why this question stayed open.
- Does permissions and row-level-access knowledge behave like the flat term or the escalator term? It is the one OI component no published benchmark ablates; EntSQL lists permission-aware querying only as future work.
- Should the Confidently-Wrong Probe's question battery be rebuilt around the five BIRD-History types, with a per-type scorecard as the client deliverable? That is a design decision, not a research question, and it belongs to the Intelligence Maturity Assessment build.
- Set a watch on Cortex Sense general-availability notes for a schema-only versus grounded contrast that holds the agent harness fixed. That single experiment is what would make 23-to-83 interpretable, and Snowflake has an incentive not to run it.
Related
- [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]] - parent brief; source of this question, and of the retirement of the 23-to-83 figure
- [[2026-09-12-pupius-four-axis-snowflake-data-ai-strategy]] - where the unpublished split was first named as the most useful open question
- [[2026-04-07-dbt-semantic-layer-vs-text-to-sql-benchmark]] - the contradicting magnitude, and the 32.7%-to-64.5% model-capability time series the escalator argument rests on
- [[2026-07-08-cortex-sense-semantic-layer-wedge-caf]] - origin of the 23/47/83 three-point version and the "context is not correctness" caveat
- [[2026-09-14-snowflake-cowork-cortex-ga-vs-preview-matrix]] - the general-availability narrowing; Cortex Sense and the plugins are not generally available
- [[2026-09-23-correctness-criticality-cannibalization-threshold]] - "83% is a ceiling on Snowflake's own data, not a floor on the client's"
- [[2026-09-23-semantic-studio-self-serve-authoring-erosion]] - the eval half of the wedge, and the governed-contract framing this brief gives a magnitude to
- [[2026-07-08-cowork-industry-plugins-vs-caf-delivery]] - where the figure entered the vault as a secondary citation
- [[concepts/semantic-layer|semantic-layer]] - approved join paths as a first-class component, now independently measured as the least schema-recoverable knowledge type
- [[2026-02-04-practical-data-modeling-dead-again-part2]] - the qualitative version of this brief's quantitative finding: agents generate perfect SQL and miss arcane business operations
Sources
Vault:
- [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]] -
~/rdco-vault/06-reference/research/2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data.md - [[2026-09-12-pupius-four-axis-snowflake-data-ai-strategy]] -
~/rdco-vault/06-reference/research/2026-09-12-pupius-four-axis-snowflake-data-ai-strategy.md - [[2026-04-07-dbt-semantic-layer-vs-text-to-sql-benchmark]] -
~/rdco-vault/06-reference/2026-04-07-dbt-semantic-layer-vs-text-to-sql-benchmark.md - [[2026-07-08-cortex-sense-semantic-layer-wedge-caf]] -
~/rdco-vault/06-reference/research/2026-07-08-cortex-sense-semantic-layer-wedge-caf.md - [[2026-09-14-snowflake-cowork-cortex-ga-vs-preview-matrix]] -
~/rdco-vault/06-reference/research/2026-09-14-snowflake-cowork-cortex-ga-vs-preview-matrix.md - [[2026-09-23-correctness-criticality-cannibalization-threshold]] -
~/rdco-vault/06-reference/research/2026-09-23-correctness-criticality-cannibalization-threshold.md - [[2026-09-23-semantic-studio-self-serve-authoring-erosion]] -
~/rdco-vault/06-reference/research/2026-09-23-semantic-studio-self-serve-authoring-erosion.md - [[2026-07-08-cowork-industry-plugins-vs-caf-delivery]] -
~/rdco-vault/06-reference/research/2026-07-08-cowork-industry-plugins-vs-caf-delivery.md - [[concepts/semantic-layer|semantic-layer]] -
~/rdco-vault/06-reference/concepts/semantic-layer.md - [[2026-02-04-practical-data-modeling-dead-again-part2]] -
~/rdco-vault/06-reference/2026-02-04-practical-data-modeling-dead-again-part2.md
Web (fetched or searched 2026-11-05):
- BIRD-History, "A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations," arXiv 2608.29345v1 (Zhou, Shi, Yang, Weng, Wu). PRIMARY-VERIFIED: Table 13 no-retriever 54.27 / retriever 60.95 / gold 67.34; Table 4 per-type retrieval F1; Tables 11 and 12 model-transfer collapse; Table 7 five-type taxonomy: https://arxiv.org/abs/2608.29345
- EntSQL, "A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge," arXiv 2606.03363v2, 6 Jul 2026 (HKUST-GZ, Alibaba Group, Guangdong Laboratory). PRIMARY-VERIFIED via paper HTML: base 4.7-6.8 across eight models, best 6.8 to 15.9 with documents to 21.4 with evidence, human experts 33.5/46.2/84.0: https://arxiv.org/html/2606.03363v2
- Snowflake, "Cortex Sense for Enterprise AI Agents" (24.1% to 86.3%; Snowflake-internal product-analytics questions; harness varied, knowledge held constant). Cited here via [[2026-09-20-cowork-plugin-accuracy-unmodeled-midmarket-data]], not re-fetched: https://www.snowflake.com/en/blog/enterprise-ai-agents-grounded-context/
- Snowflake engineering blog, Cortex Analyst text-to-SQL accuracy (90%+ on a 150-question internal suite, scoped to a single pre-joined view, no with-model versus without-model delta). Cited via the same vault brief: https://www.snowflake.com/en/engineering-blog/cortex-analyst-text-to-sql-accuracy-bi/
- Spider 2.0, "Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows," ICLR 2025. UNVERIFIED HERE: the 10.98% on external-document tasks reached this brief through a search-result summary of the paper, not a primary read: https://proceedings.iclr.cc/paper_files/paper/2025/file/46c10f6c8ea5aa6f267bcdabcb123f97-Paper-Conference.pdf
- BIRD, "Can LLM Already Serve as a Database Interface?," arXiv 2305.03111. UNVERIFIED HERE: the GPT-4 34.88% versus 54.89% evidence pair comes from secondary summaries only; primary verification is an open follow-up: https://arxiv.org/abs/2305.03111
- "An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data," arXiv 2603.07841v1. Fetched and found NOT to contain the no-evidence-collapse or "5 of 52" claims attributed to a 2026 "DIVER" paper by a search summary; recorded so the dead end is not re-walked: https://arxiv.org/html/2603.07841v1