06-reference

data engineering central duckdb polars spark serverless benchmark

2026-10-05·reference·source: Data Engineering Central·by Daniel Beach
duckdbpolarspysparkdatabricks-serverlessunity-catalogdelta-lakebenchmarkwindow-functionsjoinslakehouse

"DuckDB vs Polars vs Spark on Databricks Serverless with Joins and Window Functions" — Daniel Beach (Oct 5 2026)

Why this is in the vault

Hands-on, non-scientific benchmark (~285M-row Backblaze hard-drive-failure dataset, broadcast join + row_number/lag/rolling-average window functions) comparing PySpark-on-Serverless, DuckDB, and Polars against Unity Catalog-managed Delta tables. The clean result — PySpark-on-Serverless won, DuckDB choked on Unity Catalog write permissions and was slow once it fell back to raw S3/Delta scanning, Polars OOM'd twice and never finished — directly updates two tool-selection threads the vault already tracks on this exact stack.

The core argument

⚠️ Sponsorship

sponsored: true, sponsor_entity: Delta Lake. This is Data Engineering Central's confirmed standing sponsor (10+ issues logged in the vault project README, most recently 2026-09-14) — the newsletter runs an explicit "Thanks to Delta for sponsoring this newsletter!" block plus Beach's own disclosed daily-user framing ("I use Delta Lake daily, and I believe it represents the future of Data Engineering"). Per the standing project gotcha, treat this as commercially motivated by default. Bias implication here is narrow: the sponsorship doesn't touch the DuckDB/Polars/Spark verdict directly (Delta Lake is the storage format underneath all three engines in this test, not a competing choice), but it does mean Beach has a standing financial incentive to keep framing Delta Lake/Unity Catalog as the default lakehouse substrate rather than questioning that layer when things go wrong — notable since the DuckDB failure here is specifically a Unity Catalog permissions issue, and the piece never asks whether Unity Catalog's write-gating is itself the problem.

Mapping against Ray Data Co

Medium, not strong — this is a data-engineering benchmark, and RDCO's thesis is agent-harness engineering, not lakehouse tooling. The concrete connection is empirical continuity, not a named RDCO project: this issue directly tests the Unity Catalog write-access question that [[2026-06-07-data-engineering-central-duckdb-unity-catalog-commits]] covered four months ago (DuckDB-in-Lambda using "Catalog Commits" to get concurrent external writes to Delta tables without a Spark cluster). Beach here hits the exact READ/WRITE wall that Catalog Commits was supposed to solve — but via a plain Serverless notebook connection, not the Lambda/Catalog-Commits path — suggesting the DuckDB-writes-to-Unity story is still path-dependent and far from "just works" eight months after it first looked solved. Separately, Polars' OOM failure here reinforces the maintainer-culture/production-stability argument in [[2026-04-09-data-engineering-central-replacing-polars-with-duckdb]], where a longtime Polars advocate switched a production Lambda workload to DuckDB for the same class of reason.

The closing "humans still needed" framing gestures at RDCO's harness-engineering belief (agents are good at generating code, not at the exploratory poke-and-break work that surfaces genuinely new integration failures) but it's a one-paragraph aside, not the piece's focus — treat it as a mild reinforcement, not new evidence.

Related