"Databricks. Data Quality. MPC Server. SDK." — Daniel Beach, Data Engineering Central
Why this is in the vault
A short, explicitly-unsponsored issue where Beach points at creator Andreas Kretz's YouTube walkthrough of building a Databricks + Google Drive Data Quality Assistant via an MCP server and Zapier SDKs, using it to make a narrow claim: his own daily Databricks workflow has shifted from hand-writing SQL/Spark to instructing an agent in English and reviewing its output — a concrete, low-hype data point on how agentic tooling is actually landing in day-to-day DE work (note: issue text repeatedly typos "MCP" as "MPC").
The core argument
Beach frames Kretz's 1-hour-plus video as a rare non-hype demonstration of agentic capability inside a real data platform, rather than a "Hello World" or marketing piece. The video builds a Data Quality Assistant wired to Databricks and Google Drive using an MCP server plus Zapier's SDK. Beach's own takeaway is personal and specific: his analytics workflow changed "almost imperceptibly" over a few years — he now uses domain knowledge plus English instructions to direct an agent through research and data exploration, instead of building every dashboard or query by hand, which leaves him in a reviewer/researcher role rather than a pure implementer. His framing for skeptics: treat the agent like a junior engineer reporting to a senior — useful, but still your job to verify its output is correct, exactly as you'd check your own SQL. The named tools (MCP, LLMs, RAG, Agents) are secondary to the real question he poses: can you use these primitives to extend your own capability and solve novel problems faster.
Mapping against Ray Data Co
Beach's "senior engineer with a junior agent, verify everything" framing is the exact posture feedback_workflow_agent_output_integrity codifies for RDCO's own agent chains (pointer-not-content returns, scratchpad contamination, false "verified" stamps) — it's independent confirmation from a working DE practitioner that the failure mode isn't RDCO-specific, it's structural to any agent-as-junior-report setup. It also lands squarely on RDCO's own MCP surface area: the Kretz demo chains an MCP server + Zapier SDK to a data platform the same way this very pipeline chains Gmail/Notion/graph-ingest MCP tools to the vault — a useful outside data point that "MCP server wired to a real platform" is becoming an ordinary practitioner pattern, not a frontier one.
Related
- [[2026-07-22-data-engineering-central-agentic-de-hugo-lu]] — same author's higher-level framing of agentic DE's "closing the loop" problem; this issue is the ground-level, single-practitioner version of that same shift.
- [[2026-08-10-data-engineering-central-quasi-agentic-pipelines-databricks-airflow]] — same author, same "use tools you already have, verify non-deterministic output" instinct, applied to Airflow/Databricks pipelines instead of a Kretz-built assistant.
- [[feedback_workflow_agent_output_integrity]] — the RDCO-side codification of the same verify-the-junior-agent discipline Beach argues for here.