Semantic layer + validation controls — Anthropic's approach, RDCO's status, and a portable provenance-footer spec
Founder follow-up (2026-06-03) to [[2026-06-03-anthropic-self-service-data-analytics-with-claude]]: how did they structure the semantic layer / how would we; do we have the three controls (offline evals, adversarial-review subagent, provenance footers); and the provenance footer for his phData copilot. Anthropic specifics are grounded in the blog note; the build-guidance + RDCO audit are my synthesis. Anthropic named NO warehouse/BI vendor — the named tools below ("how you'd build one") are mine, not theirs.
Q1 — The semantic layer: how Anthropic structured it, and how you'd build one
What a semantic layer is (general): a governed mapping from raw tables to business concepts — canonical metric definitions, the entities/dimensions you can slice by, and the approved join paths to compute them. The point: a query becomes "net revenue by acquisition cohort, last quarter" instead of a fragile 6-table join with someone's private definition of "revenue." Established implementations: dbt Semantic Layer / MetricFlow, Cube, LookML, or Snowflake semantic views / Cortex Analyst semantic model.
How Anthropic structured it (4 layers, per the blog):
- Data Foundations — dimensional modeling, canonical datasets, metadata, freshness checks.
- Sources of Truth — the semantic layer itself: metric definitions + lineage/transformation graph + a query corpus (known-good example queries) + business context.
- Skills — Markdown procedural-knowledge files the agent reads on demand. These act as a router: the agent is structurally required (by skill instruction) to consult the semantic layer FIRST, falling back to reference docs. That narrows the search space from "a million-field warehouse" to "a few dozen curated files." This is the load-bearing trick — not a smarter model, a required reading order.
- Validation — offline evals + online controls (see Q2).
The discipline that makes it hold: skills live in the same repo as the transformation code, and CI flags any data-model change that doesn't also touch a skill file → ~90% of data PRs now carry a skill change. The semantic layer doesn't rot because changing the model without updating its procedural doc fails CI.
How you'd build one (RDCO or phData):
- Define canonical metrics + entities + approved join paths in one governed spec (dbt SL / Cube / Cortex Analyst semantic model).
- Write a small set of skill docs = "to answer questions about X, use metric
m, sources, these analysis patterns; never query raw tabletdirectly." - Force tier order: agent must hit the semantic layer first, reference docs second, raw warehouse last (or never).
- Colocate skills with transform code + CI-gate so the docs can't drift from the model.
- Result Anthropic claims: eval accuracy ~21% → consistently >95% purely from skills + semantic-layer-first routing.
Q2 — Does RDCO have the three controls?
| Control | RDCO status | Detail |
|---|---|---|
| Adversarial-review subagent | ✅ Yes — strong | This IS RDCO's existing fresh-eyes/verify pattern: /verify-vault-write, /verify-strategic-output, /verify-dispatch, /design-critic, /video-critic, /verify-pdf-output, plus the dynamic-workflows adversarial-verify pattern. Used twice on 2026-06-03 alone (verifying the ant CLI commands + the Opus-4.8 fabrication claim against primary sources). Anthropic's "adversarial-review subagent" = a thing RDCO already does by default. Note their cost data: +~6% accuracy for +32% tokens / +72% latency — so reserve it for high-stakes outputs, which is already RDCO's posture. |
| Offline evals | 🟡 Partial | RDCO has the deterministic newsletter invariant audit (audit-newsletter-outputs.py, zero LLM calls) — a genuine offline eval over outputs, gated post-condition. But it's domain-specific (newsletter shape), not a general "did the agent answer correctly" eval suite. Building blocks present; a broad offline-eval harness is not. |
| Provenance footers | 🟡 Building blocks, not formalized | RDCO vault notes already carry provenance in frontmatter: source, source_url, source_fidelity (incl. the reconstructed-from-web marker), date. That IS provenance metadata — but it's per-document, not a per-answer output footer with source-tier + freshness + owner. The convention exists; formalizing it as an output footer is the gap. |
Honest read: RDCO is strongest exactly where Anthropic spent the most (adversarial review). The two partials (offline evals, provenance footers) are where RDCO has the raw materials but not the formalized system. Neither is a heavy lift to close.
Q3 — A portable provenance-footer spec (for the phData copilot AND RDCO outputs)
The key insight from Anthropic: a provenance footer does not fix silent failure (plausible-but-wrong answers). What it does is give the human the metadata to catch it — it converts an opaque answer into one a person can sanity-check in five seconds. Even a minimal version materially raises trust.
Minimal spec (drop-in):
— Provenance —
Source: <semantic-layer metric `revenue_net` | table `fct_orders` | doc `pricing.md`>
Tier: certified | derived | ad-hoc ← how trustworthy the source is
Freshness: data as of <timestamp>; pipeline last ran <time>
Owner: <team / person who owns the source>
Confidence: high | medium | low — <one caveat, e.g. "excludes refunds" / "long-tail question, not eval-covered">
Design notes:
- Tier is the highest-value field. "certified" = a governed semantic-layer metric; "derived" = computed on the fly from canonical tables; "ad-hoc" = the agent improvised a query. A user treats a
certifiedanswer differently from anad-hocone — that single badge does most of the trust work. - Freshness catches the most common real-world wrong-answer: stale data. Cheap to populate (pipeline run timestamps), high catch rate.
- The confidence caveat is where the agent flags its own uncertainty — "this was a long-tail question my eval set doesn't cover" is exactly the silent-failure early-warning.
- Make it cheap. A footer the agent populates from metadata it already has (source name, table freshness, owner from the catalog) costs near-zero tokens. Don't make it a second LLM pass.
- Structured output is for the FORMAT, not the truth (load-bearing). The Anthropic-API structured-output feature (JSON-schema / tool-forced output) is a clean fit for assembling the footer into a guaranteed-valid object — every field present, enums enforced. But structured output constrains shape, not content: the model will emit
tier: "certified"+ a plausible timestamp while hallucinating both. A well-formed fabricated footer is worse than none (false confidence) — the silent-failure trap with a nicer wrapper. So: source / tier / freshness / owner must come from deterministic lookups (semantic-layer catalog + pipeline run metadata) via tool calls; structured output only assembles those known-true values + the model's own confidence/caveat (the one field the model legitimately owns). Rule: tool use grounds the values, structured output formats them. (Founder Q, 2026-06-03; ties to the Opus-4.8 fabrication failure mode in [[2026-06-03-every-opus-4-8-gets-in-your-way]].) - For the phData copilot specifically: even just
Source + Tier + Freshnesson every answer is a strong v1. It's the difference between "the copilot said revenue was X" and "the copilot said revenue was X, from a certified metric, data as of this morning" — the second is something a stakeholder can act on.
Related
- [[2026-06-03-anthropic-self-service-data-analytics-with-claude]] (the source)
- [[2026-06-03-claude-api-sdk-capability-map-rdco-use-cases]] (the broader capability map)
- [[2026-04-19-mac-vs-published-data-quality-frameworks]] (MAC — the validation layer is the seed MAC doesn't yet cover)