"DuckDB vs Polars vs Spark on Databricks Serverless with Joins and Window Functions" — Daniel Beach (Oct 5 2026)
Why this is in the vault
Hands-on, non-scientific benchmark (~285M-row Backblaze hard-drive-failure dataset, broadcast join + row_number/lag/rolling-average window functions) comparing PySpark-on-Serverless, DuckDB, and Polars against Unity Catalog-managed Delta tables. The clean result — PySpark-on-Serverless won, DuckDB choked on Unity Catalog write permissions and was slow once it fell back to raw S3/Delta scanning, Polars OOM'd twice and never finished — directly updates two tool-selection threads the vault already tracks on this exact stack.
The core argument
- Beach's stated hypothesis going in: Databricks Serverless compute should let lightweight engines (DuckDB, Polars) beat Spark on medium workloads, since both claim first-class Unity Catalog support. Result: the opposite happened.
- DuckDB: Unity Catalog connector requests READ/WRITE and gets rejected because Serverless only grants READ (he suspects this is a Spark Connect API artifact, not a DuckDB bug) — a dead end even for a pure SELECT query. Recreating the managed tables as "External" tables to get an S3 URI worked around the catalog block, but DuckDB scanning Delta directly off S3 (
delta_scan) was "slower than expected," just not fatal. - Polars: connected to Unity Catalog fine (further than DuckDB got) but died ~8 minutes in with no clear error; a second attempt crashed the cluster; a third attempt with explicit column pruning hit
SIGKILL/exit 137 (OOM). Beach frames this as a recurring, long-standing Polars pattern on anything beyond small data, not a one-off. - PySpark-on-Serverless finished cleanly and fastest, undercutting the premise of the whole experiment.
- Closing frame: even in an "agentic coding" era, this kind of tool-integration friction (catalog permissions, OOM edge cases) still requires a human willing to poke, break, and retest — explicitly contrasted against agents, which Beach argues are trained on the past and not well-suited to probing genuinely new failure modes.
⚠️ Sponsorship
sponsored: true, sponsor_entity: Delta Lake. This is Data Engineering Central's confirmed standing sponsor (10+ issues logged in the vault project README, most recently 2026-09-14) — the newsletter runs an explicit "Thanks to Delta for sponsoring this newsletter!" block plus Beach's own disclosed daily-user framing ("I use Delta Lake daily, and I believe it represents the future of Data Engineering"). Per the standing project gotcha, treat this as commercially motivated by default. Bias implication here is narrow: the sponsorship doesn't touch the DuckDB/Polars/Spark verdict directly (Delta Lake is the storage format underneath all three engines in this test, not a competing choice), but it does mean Beach has a standing financial incentive to keep framing Delta Lake/Unity Catalog as the default lakehouse substrate rather than questioning that layer when things go wrong — notable since the DuckDB failure here is specifically a Unity Catalog permissions issue, and the piece never asks whether Unity Catalog's write-gating is itself the problem.
Mapping against Ray Data Co
Medium, not strong — this is a data-engineering benchmark, and RDCO's thesis is agent-harness engineering, not lakehouse tooling. The concrete connection is empirical continuity, not a named RDCO project: this issue directly tests the Unity Catalog write-access question that [[2026-06-07-data-engineering-central-duckdb-unity-catalog-commits]] covered four months ago (DuckDB-in-Lambda using "Catalog Commits" to get concurrent external writes to Delta tables without a Spark cluster). Beach here hits the exact READ/WRITE wall that Catalog Commits was supposed to solve — but via a plain Serverless notebook connection, not the Lambda/Catalog-Commits path — suggesting the DuckDB-writes-to-Unity story is still path-dependent and far from "just works" eight months after it first looked solved. Separately, Polars' OOM failure here reinforces the maintainer-culture/production-stability argument in [[2026-04-09-data-engineering-central-replacing-polars-with-duckdb]], where a longtime Polars advocate switched a production Lambda workload to DuckDB for the same class of reason.
The closing "humans still needed" framing gestures at RDCO's harness-engineering belief (agents are good at generating code, not at the exploratory poke-and-break work that surfaces genuinely new integration failures) but it's a one-paragraph aside, not the piece's focus — treat it as a mild reinforcement, not new evidence.
Related
- [[2026-06-07-data-engineering-central-duckdb-unity-catalog-commits]] — the Catalog-Commits pattern this issue's Unity Catalog failure directly complicates
- [[2026-04-09-data-engineering-central-replacing-polars-with-duckdb]] — prior production-stability verdict against Polars that this issue's OOM failure reinforces
- [[2026-06-22-data-engineering-central-datafusion-comet-spark]] — same author, same "clever tooling, rough developer experience" verdict pattern applied to a different Spark-adjacent engine