Microsoft Copilot Studio for contract-citation RAG — legibility, citation architecture, and eval design

Built 2026-06-03 to support a first-time Copilot Studio engagement: a conversational, retrieval-only agent over PDF contracts in Box, business cases = general Q&A + contract comparison + document/page/paragraph citation. Two product questions were web-researched against current docs; eval design + the addendum-modeling pattern are synthesis.

1. Is Copilot Studio "legible to an agent" / programmatically buildable?

Yes — meaningfully more than its low-code reputation suggests. It is NOT GUI-only.

• The real "as-code" loop: pac copilot clone (agent → local YAML) → edit topics / instructions / knowledge / tools / triggers in VS Code (GA extension with schema validation) → pac copilot push → pac copilot pack (into a Dataverse solution) → ALM pipeline → pac copilot publish. Commands: init, create, clone, pull, push, pack, publish, extract-template, delete, list, status. (pac copilot reference)
• Source control + CI/CD: agents are Dataverse solution components; standard ALM (Power Platform Pipelines, Azure DevOps, GitHub Actions, Git source control). (ALM strategy, export/import via solutions)
• Microsoft shipped a Claude Code skill for it: microsoft/skills-for-copilot-studio — a plugin for Claude Code / Copilot CLI / VS Code that authors Copilot Studio agents as YAML (topics, actions, knowledge, triggers, variables, instructions) with schema validation, on top of pac copilot. Labeled experimental / not officially supported; YAML schema may change. This is the direct answer to "is it legible to an LLM agent": Microsoft itself built the harness.
• MCP is GA in Copilot Studio (GUI wizard: Tools → Add a tool → MCP; or as an OpenAPI YAML custom connector carrying x-ms-agentic-protocol: mcp-streamable-1.0, which travels with ALM). Transport: Streamable HTTP only (SSE removed after Aug 2025). (connect existing MCP server)

What's NOT as-code (GUI-locked / absent):

• No clean public REST API for full agent/topic/knowledge authoring CRUD. (The "Copilot Studio REST API" docs are about the agent calling a REST tool, not authoring the agent.) Don't assume a REST authoring endpoint.
• No Bicep/ARM/Terraform resource that authors the agent itself. Terraform's microsoft/power-platform provider stops at environment / solution / App-Insights scope. IaC pattern is composite: Terraform/azd provisions env + Azure deps, then pac deploys the agent.
• Final publish + some post-import config (auth, channels, icon) commonly route through the platform/UI.
• Solution-export gotcha (teach this): "Not all agent components are included — custom topics and knowledge sources might not be included" unless you explicitly Add required objects. Managed solutions can't be re-exported → keep an unmanaged source. Topic names with . can't export.

Distinct, adjacent product: the Microsoft 365 Agents SDK (evolution of Bot Framework) is pro-code, custom-orchestration, any-model — a different build path that interops with Copilot Studio but isn't the same artifact. If a fully code-defined agent is wanted, that's the SDK; you then lose the low-code connectors/topics/knowledge surface. (Agent Builder vs Copilot Studio)

2. Citation fidelity + the RAG architecture decision

Copilot Studio's native grounding is real RAG but citations are document-level, not page/paragraph.

• It ingests files into Dataverse, chunks + vector-indexes them, retrieves relevant chunks at query time. PDF supported (incl. text-in-images). (knowledge-unstructured-data, knowledge-copilot-studio)
• BUT the documented citation unit surfaced to the user is the source document / link — no Microsoft Learn statement that it exposes page-number or paragraph/chunk citations. Treat native page/paragraph citation as unverified — docs don't support it.
• Citations can't be used as inputs to other tools/actions — you can't programmatically post-process them.
• No accuracy check; non-deterministic answers ("the system doesn't perform an accuracy check"; same question → varying answers). For binding-contract Q&A where a wrong citation is a liability, those two facts are the headline. (faqs-generative-answers)
• Limits (load-bearing for a contract corpus): 500 knowledge objects/agent, 5 sources at a time, PDF up to 512 MB with Work IQ semantic index, ~4–6 h sync latency.

Connecting Box — two paths, neither gives native chunk-cite:

• First-party Box Power Platform connector = automation-grade (upload/get/search file ops), NOT a vector knowledge source. Box isn't in Copilot Studio's supported-knowledge-source table.
• Box hosted remote MCP server (mcp.box.com, GA Aug 2025, OAuth) = cleaner fit, but it attaches as tools, not a vector knowledge source — so citation quality depends on what Box AI returns (whether it yields page/clause citations is undocumented → must test the tool output). box.com lists Copilot Studio as supported, but Box's GA support article didn't name it — verify hands-on.

Recommendation (his instinct is correct): for page/paragraph contract citation, bolt on a dedicated retrieval/RAG layer; use Copilot Studio as the conversational shell.

• Snowflake Cortex Search is a strong fit: retrieval-only, returns the verbatim matched chunk + any metadata columns you request (DOC_ID, SOURCE_URL, PAGE, CLAUSE). You own chunk boundaries + provenance. Recommended chunk ≤512 tokens. Exposable to Copilot Studio as an MCP tool (Snowflake Cortex Agents MCP). (cortex-search-overview)
• The load-bearing nuance: Cortex Search (or any retriever) returns whatever you indexed. Page/paragraph granularity is created by your parsing pipeline at ingest (e.g. AI_PARSE_DOCUMENT or a PDF parser that emits page/clause fields) — NOT by the search engine. The citation is only as good as the metadata you populate. This is the part to own.
• Architecture: Box (source of truth for PDFs) → parsing pipeline (page/clause-tagged chunks) → Cortex Search index → Copilot Studio chat shell calls it as an MCP tool → provenance footer built from the chunk's metadata. (Ties to [[2026-06-03-semantic-layer-validation-controls-rdco]]: tool grounds the citation values, the chat layer formats them.)
• When native grounding is enough: lower-stakes "find the right doc + summarize," document-level link acceptable, corpus under the 500-object/5-source limits, content in SharePoint/OneDrive/Dataverse. For binding contract clause-citation, go external-RAG.

3. The addendum problem (the sharpest modeling wrinkle)

Signed-final contracts with post-execution addendums that amend language or add sections. This breaks naive RAG and is the highest-value thing to get right.

• Model a contract as original ⊕ ordered addendums, not a single document. A clause's current effective state = original, overridden by later addendums touching that clause.
• Ingest-time metadata is the fix: tag every chunk with contract_id, is_addendum, amends_contract_id, effective_date, affected_sections. Without addendum-linkage metadata, the retriever can't know a clause was superseded.
• "Current effective X" questions are the failure mode: a naive retriever pulls the original clause and misses the amending addendum (or vice versa). Answering requires retrieving the original clause AND any later addendums to it, then reasoning about supersession.
• Citation implication: an addendum-modified answer must cite both sources — original §X and the amending addendum dated Y. The provenance footer should carry both.

4. Eval design (grounding evals) for contract RAG

He needs a robust question list for grounding evals. Structure each item: question → expected answer (ground truth) → expected citation(s) (doc + page/clause) → category. Measure three things separately: retrieval (right chunk pulled?), answer faithfulness (correct + grounded, not hallucinated?), citation accuracy (right place cited?). This is the offline-eval discipline from [[2026-06-03-anthropic-self-service-data-analytics-with-claude]], applied.

Four categories (build ~5–15 each, co-created with the client team — that's itself a great shadowing deliverable):

1. General Q&A (single-doc factual): parties / grantor-grantee; term & duration; territory/countries covered; rights granted (sync, master use, mechanical, performance); exclusive vs non-exclusive; royalty rate / fee / advance by role (composer / songwriter / producer); effective + signature dates; usage restrictions; credit/attribution obligations; reversion/termination conditions.
2. Contract comparison (cross-doc): compare territory grants for the same work across country-specific contracts (where do they differ?); which agreements include a reversion clause and which don't; for one title, decompose all contracts and what rights each covers; which role-equivalent contract has the most favorable rate; do producer vs composer agreements have overlapping/conflicting grant scopes; which contracts are silent on a topic (e.g. streaming / AI usage).
3. Citation-precision (the provenance eval — the hard one): "What does the contract say about [topic], and exactly where (section/page)?"; "Which paragraph defines 'Territory'?" — these directly test page/paragraph citation fidelity, the thing native Copilot Studio likely fails.
4. Addendum-aware (the gotcha category — include these deliberately): "What is the current effective royalty rate for [work]?" (requires original ⊕ addendum); "Has the term been extended — by what, and when?"; "Which clauses of the original were superseded or amended?"; "Is this clause still in force or replaced?" — and require the answer to cite both original and amending addendum.

Methodology note for the shadowing sessions: co-build the eval set with the client (it doubles as eval-driven-development training); run the same set against each architecture variant (native Copilot Studio grounding vs external-RAG) to show the citation-fidelity gap with data rather than assert it — exactly the 21%→95% style before/after.

Honest uncertainties (verify hands-on before architecting around them)

1. No published Microsoft figure for Copilot Studio citation faithfulness / hallucinated-citation rate — qualitative limits only.
2. Whether Box AI (via Box MCP) returns page/clause-level citations is undocumented — test the actual tool response.
3. "Copilot Studio is an officially named Box MCP integration" is on box.com's product page but not confirmed in Box's GA support article — verify.
4. Copilot Studio knowledge/MCP features and limits (500 objects, 512 MB, transport) move fast — recheck near build time.

Related

• [[2026-06-03-semantic-layer-validation-controls-rdco]] (provenance footers, tool-grounds-values, offline evals)
• [[2026-06-03-anthropic-self-service-data-analytics-with-claude]] (eval discipline + silent-failure)
• [[2026-06-03-claude-api-sdk-capability-map-rdco-use-cases]] (structured output, MCP, the broader surface)