Data Engineering Weekly #270 — @Ananth Packkildurai (May 18 2026)

Why this is in the vault

First DEW issue since #267 to land in the vault (268 and 269 were not filed). Mix this week is heavy on infra / engine convergence: an Airbnb governance piece (Viaduct GraphQL mesh), a Netflix identity piece (Data Projects replacing user-bound ACLs with team-owned application identities), two engine-shape pieces (Eric Sun's Query Proxy + Alibaba MySQL-on-DuckDB HTAP), and a useful DuckDB FTS exploration. Three sponsor placements detected — including an Altimate Code "agents for data engineering" pitch that is on-thesis for the [[concepts/products-for-agents|products-for-agents]] argument and worth tracking even though we are not following the link.

Two third-party deep-fetches this week (per skill cap): Eric Sun's Query Proxy and Pete's DuckDB FTS — the two pieces with the strongest mapping to active RDCO threads (agent-facing data abstraction + vault-side DuckDB tooling).

Sponsorship

Three paid / self-promo placements detected:

1. Top-of-issue: "The Data Platform Fundamentals Guide" — Dewpeche eBook (Ananth's own). Free download. Counts as self-promo, not third-party sponsor. Format: lead item before the curated section starts. Disclose as author cross-promo, not paid sponsor.
2. Mid-issue (slot 1): "Sponsored: Agents for Data Engineering" — Altimate Code. Open-source MCP-style tool layer — "100+ deterministic tools for SQL, lineage, dbt, and warehouse connectivity," claims #1 on something called ADE-Bench. Tech-stack agnostic. This is the second time in three months we've seen vendors explicitly positioning around an "agent tool layer for data engineering" thesis (after the Rill Metrics SQL piece [[2026-04-08-rill-metrics-sql-semantic-layer-agents]]). Worth tracking the category even though the sponsor placement itself is biased.
3. Mid-issue (slot 2): "Sponsored: Free Course - AI-Driven Data Engineering" — Dagster University. Recurring slot, same as #267. Pitches building a production-ready ELT pipeline from prompts using Dagster + agentic coding workflows.

Neither paid placement appears to bias the editorial picks. The recurring Dagster slot is fully normalized into the issue layout now — same observation as #267, no change.

Mapping against Ray Data Co

This is a medium RDCO-relevance week. Two pieces map onto active threads, the rest are file-for-reference.

1. Eric Sun's Query Proxy is a clean architectural pattern for the "agent-facing data layer" thesis. Async query API (return query ID, fetch later), presigned Parquet URLs (30MB chunked, ZSTD-compressed) as the result-delivery contract, hierarchical RBAC (service-credential → DB-credential → object-RBAC), and federation across hot Postgres + warm Iceberg with a configurable cutoff. The architectural shape is exactly what an agent needs when it's reasoning over a heterogeneous warehouse: a single async API, results that land in object storage so downstream Polars/DuckDB can pick them up, and quota enforcement at the query-shape level so a runaway agent can't tank the warehouse. Pairs cleanly with [[2026-04-08-rill-metrics-sql-semantic-layer-agents|Rill Metrics SQL]] (semantic layer for agents) and [[2026-03-31-semistructured-data-layer-does-the-work|Natkins on the data layer]] — Query Proxy is the transport shape, Metrics SQL is the semantic shape, and Natkins's piece is the thesis. File this triplet together when the agent-deployer toolkit comes up in phData consulting context.

2. Pete's DuckDB FTS piece is directly relevant to the vault's own architecture. [[../01-projects/graph-db-eval/vertex-edge-dictionary.md|The vault graph is already on DuckDB]]. The FTS extension is a single INSTALL fts; LOAD fts; away from giving qmd / graph-query a BM25 keyword path that doesn't require Elasticsearch or a separate process. Tunable k₁ (term-frequency weight) and b (length normalization) give us enough knobs for "exact-term vault lookup" without bolting on infra. Limitations are real — no ts_headline-style highlighting, weak phrase support, no synonym dictionary — but for vault-scale exploratory text search those are acceptable. Action: file as a candidate evaluation for the qmd lex-search path; if BM25 quality improves over the current ranker it's a near-zero-cost win.

3. Netflix's Data Projects piece (durable team-owned identities replacing per-user ACLs) is reference-quality for any future RDCO writing on multi-agent service-identity patterns. Agents need durable identities for the same reason team-owned pipelines do — the alternative is access breakage every time the agent that owns a workflow gets rebuilt. The Netflix framing is the cleanest public articulation of this pattern. File for the agent-deployer toolkit; not urgent.

4. Airbnb Viaduct is a federated-GraphQL mesh story — interesting as a counterpoint to the data-mesh-as-decentralized-pipelines framing the data-engineering world has settled on. Viaduct = one global schema + independently contributed resolvers, which is closer to the original Zhamak Dehghani "domain-oriented data products" framing than what most data-mesh implementations have become. Worth holding alongside [[2026-04-27-data-engineering-weekly-issue-267|Monzo "Interfaces"]] as a different shape of the same governance instinct.

The Faire / Meta / Databricks / Alibaba / Marc Bowes pieces are reference-only. Marc Bowes on Aurora DSQL Coupler is architecturally interesting (CDC as a fan-out subscriber on an already-decoupled journal) but RDCO-tangential. Alibaba's MySQL-on-DuckDB HTAP is fun to know (and is part of the broader story that DuckDB is becoming a serious warehouse-style engine, which is consistent with the [[2026-04-09-data-engineering-central-replacing-polars-with-duckdb|Daniel Beach piece]] from April).

Issue contents

Eight curated items + three sponsor/self-promo placements. Mix this week is engine convergence (warehouses, GraphQL, CDC, HTAP) + governance (Airbnb mesh, Netflix identities) + one search-ranking case study.

1. Airbnb — Viaduct 1.0 and the future of Airbnb's data mesh (medium / airbnb-engineering). Viaduct is a multi-tenant GraphQL federation system — one global schema, independent teams contribute schema + resolvers. Different shape from what data-mesh has become in data engineering, closer to original Zhamak framing.
2. Netflix — Data Projects: Managing Data Assets at Netflix Scale (medium / netflixtechblog). Replaces per-user "On-Behalf-Of" identities with durable team-owned application identities. ACL breakage on team-change / departure motivates the redesign. Strong governance case study.
3. Meta — Migrating Data Ingestion Systems at Meta Scale (engineering.fb.com). Shadow-testing-mode migration of tens of thousands of petabyte-scale pipelines. Partition-level "bad" flags halt new delta landings and force merge with last-known-good — clever blast-radius limiter.
4. Databricks — The Convergence of Open Table Formats and Open Catalogs: Catalog Commits GA (databricks.com/blog). Unity Catalog + Delta Lake catalog-commit GA. Notes the metastore-lifecycle arc: Hive → S3-as-metastore → back to catalogs.
5. Eric Sun — A Query Proxy for Analytical and Fast Data (eric-sun.medium.com). [DEEP-FETCHED — see notes below.] gRPC proxy fronting StarRocks, VeloDB, RisingWave, ClickHouse, Snowflake, Databricks, Iceberg. Async query API, presigned Parquet URL result delivery, hierarchical service-credential RBAC, Postgres ↔ Iceberg federation across a configurable cutoff, query-shape telemetry + quotas. Strong RDCO match.
6. Faire — How we rebuilt search ranking at Faire with deep learning (craft.faire.com). XGBoost → Deep and Cross Network with session-normalized listwise cross-entropy. New rankers (Brand Page) fine-tuned from Product Search model — cuts new-surface launch cycles by 50%. +2.14% order volume NA, +1.54% EU. Multi-task representation as the reuse mechanism.
7. Alibaba — When MySQL Meets the Columnar Storage Engine DuckDB in the AI Era (alibabacloud.com/blog). AliSQL integrates DuckDB as a pluggable storage engine to give MySQL HTAP behavior. Ananth's editorial note: this fits the broader pattern of DuckDB drifting toward warehouse-style use (Motherduck client-server protocol referenced as evidence).
8. Marc Bowes — Aurora DSQL: Meet Coupler (marc-bowes.com). CDC subscriber on top of DSQL's already-decoupled journal fan-out. Architecturally clean because the read/write path is already split — CDC is just another reader, not a load-causing tap on the writer.
9. Pete — Full-Text Search with DuckDB (peterdohertys.website). [DEEP-FETCHED — see notes below.] INSTALL fts; LOAD fts; BM25 with tunable k₁ / b. Validates on 13k email-archive use case. Limits: no highlighting, weak phrase support, no synonym dictionary. Ananth flags Apache Pinot and ParadeDB as the other systems that try to do both free-text + analytical.

Curation section — per-link notes

Deep-fetch notes

Eric Sun — Query Proxy. Engines fronted: StarRocks, VeloDB, RisingWave, ClickHouse, Snowflake, Databricks, Iceberg/Delta. RBAC is hierarchical: service-credential → DB connection → DB credential → data-object RBAC. Object-level perms managed in the existing data-access systems; proxy handles credential→connection mapping. Query API is async — client gets a query ID immediately and is notified when results are ready. Results delivered as presigned S3/GCS URLs to a set of ~30MB Parquet files (ZSTD or LZ4 compression, with ZSTD winning end-to-end latency despite slower compression because of smaller file sizes). Federation: when a query spans a configured time cutoff, the proxy fires separate queries against hot Postgres and warm Iceberg / Delta and unifies. Observability: tracks query shapes / volumes / latencies; enforces query-shape-level quotas on time, bytes-scanned, bytes-spilled; can suggest derived-table creation for expensive repeated query shapes. Net pattern: the right architectural shape for an agent-facing data layer — single async interface, results land in object storage so downstream tools (Polars / DuckDB / direct file readers) can take over without going back through the warehouse, and quotas guard against runaway agents at the query-shape level rather than the credential level. File alongside Rill Metrics SQL as the transport-layer complement to Metrics SQL's semantic layer.

Pete — DuckDB FTS. Install/use: INSTALL fts; LOAD fts; then PRAGMA create_fts_index to define the index over chosen columns. Stemming + stop-word handling parameterized. Scoring: Okapi BM25 with tunable k₁ (term-frequency weight: lower = presence-binary, higher = repeated-mention reward) and b (length normalization: 0 = ignore doc length, 1.0 = penalize long docs). Validated on a 13,010-message email archive — exploratory text mining and document-archive analysis. Memory + import progress are transparent during indexing. Limitations vs mature engines: no query-term highlighting (Postgres has ts_headline, DuckDB does not), limited phrase-query support, no vector-search or synonym-dictionary plugins. Positioning: "good enough for most exploratory use cases" with a documented migration path to Postgres tsvector or Elasticsearch if requirements grow. Net for the vault: the [[../01-projects/graph-db-eval/vertex-edge-dictionary.md|vault graph is already on DuckDB]], so adding the FTS extension is near-zero-cost. Action candidate: evaluate as a BM25 keyword path inside qmd lex-search; if quality improves over current ranker it's a clean win without adding infrastructure.

Cross-promo check

One self-promo placement (Dewpeche eBook, top-of-issue) — disclosed above. Two paid sponsor placements (Altimate Code, Dagster University) — disclosed above. Ananth does not appear as author on any curated link. No covert self-cross-promotion.

Related

See related: block in frontmatter.