Data Engineering Weekly #295
Why this is in the vault
Meta's internal AI coding assistant (DEmate) and two companies' shared-definitions chatbots (Mercury, Klaviyo) are live field evidence for the "agent + local institutional knowledge" pattern RDCO already builds on; worth tracking against our own sub-agent conventions.
The core argument
Packkildurai's editor's note frames analytical engineering as the discipline that adopted AI earliest, having historically done two jobs: ad-hoc "tap the shoulder" Q&A (a weak precursor to today's AI agents) and building standard metrics for cross-team reuse. His open question is how AI agents augment — or absorb — those two functions next.
Curation section
- PostHog — rolled back a DuckLake-based warehouse product in favor of Trino; argues modeling data is the harder problem than storage format, and questions whether DuckLake repeats old Hive Metastore mistakes.
- Uber — "Making Every Core Count" on faster Apache Spark jobs; Packkildurai speculates Gluten/Velox-based execution could displace Spark within a couple of years.
- Sponsored — StarTree: benchmarks Trino, ClickHouse, and Pinot against the same 12.2B-row Iceberg/Parquet dataset, showing bottlenecks shift between CPU, storage reads, and coordination by workload.
- JetBrains — embedded DuckDB inside DataGrip to let users run cross-database queries via custom table functions.
- Meta — "F3 Lightning" lets ads ML feature teams release independently while still sharing underlying computation, avoiding one faulty feature blocking the whole pipeline.
- Sponsored — Astronomer: Nov 18 webinar ("WTF is happening in data engineering?") on AI agents writing/fixing pipelines and when DAG-based orchestration still applies.
- Mercury — built a Slack bot on shared metric definitions to clear a growing backlog of ad-hoc data questions; the team now uses bot queries to find and fix gaps in underlying data.
- Klaviyo — similar pattern: an AI tool answering analytics questions against trusted definitions and query checks, built specifically so a small data team could stop being the bottleneck.
- Ka Ling Wu (Medium) — reviews semantic-layer/text-to-SQL accuracy research; most studies conflate "explained business context" with "enforced metric definitions," and sample sizes are small — document meaning, discount headline accuracy claims.
- Meta — DEmate: an internal coding assistant purpose-built for Meta's own data infrastructure, using task-specific instructions, internal examples, and automated test checks to guide code changes where general-purpose coding tools struggled.
- David G. Smith — proposes combining searchable catalogs with Iceberg's versioned tables for spatial/geographic data, removing the need for a dedicated server per dataset.
- Quince — built shared infra for training, batch/real-time serving, and monitoring to make ML deployment repeatable instead of bespoke per model.
Mapping against Ray Data Co
Meta's DEmate write-up is the concrete connection: a large engineering org building an internal coding agent around task-specific instructions, internal examples, and automated checks — because general-purpose coding tools failed on their custom infrastructure — is the same shape as RDCO's own IMPLEMENTATION-NOTES sub-agent pattern and the station-based skill brigade (spec-author → test-author → code-author → critic). It's independent validation, at Meta's scale, that bespoke scaffolding beats a bare prompt-and-parse loop for in-house engineering work — the same conclusion behind routing typed judgments to Jev instead of ad hoc LLM parsing. Separately, Mercury and Klaviyo's shared-definitions chatbots are a live version of the "tap the shoulder" function the editor's note describes being absorbed by agents — directly relevant to how any future RDCO-built internal data agent should be scoped (answer from enforced definitions, not freeform SQL generation), and a caution flag given Ka Ling Wu's point that headline text-to-SQL accuracy claims often hide weak evaluation.
⚠️ Sponsorship
Two sponsored blocks this issue: StarTree (Trino/ClickHouse/Pinot-on-Iceberg benchmark) and Astronomer (Nov 18 "WTF is happening in data engineering?" webinar). Both are vendor-paid placements inline with editorial content, continuing the pattern from issue #294 (Astronomer + PhoenixAI) — Astronomer has now sponsored at least two consecutive issues; treat both blocks as promotional, not independent benchmarking or commentary.
Related
- [[2026-11-02-data-engineering-weekly-294-agent-governance-controls]]
- [[2026-10-26-data-engineering-weekly-293-semantic-metric-layer-race]]
- [[2026-09-28-data-engineering-weekly-289-jev-pipelines-data-agents]]
- [[2026-04-04-dbt-agent-skills]]
- [[feedback_implementation_notes_sub_agent_pattern]]
- [[project_jev_for_typed_set_problems]]