"Polars 2.0 vs DuckDB 2.0" — Daniel Beach (Nov 2 2026)
Why this is in the vault
Beach re-runs his signature stress-test (undersized compute, oversized data, same Backblaze hard-drive-failure CSV dataset he used in the Oct 5 Databricks/Unity-Catalog benchmark) against the fresh 2.0 releases of both tools, and gets a sharper, more mechanistic answer than before: DuckDB 2.0 finishes a 23GB aggregation on a 2GB box in 14 minutes via its new spill-to-disk/async S3 I/O, while Polars 2.0 OOMs on the identical job at 2GB, 4GB, and 8GB — because Polars 2.0's advertised "default spill-to-disk" explicitly excludes group-by and window operations (confirmed by quoting Polars' own release notes). A coda one week later: Polars 2.1 adds group-by/window spill and then beats DuckDB 2.0 by 2x on the same job — the fastest a dataframe-tool stability gap has closed in this tracked thread.
The core argument
- Polars 2.0 release highlights per Beach: streaming engine on by default, spill-to-disk on by default (but not for group-by/join — the exact operation his benchmark needs), SQL as a first-class API (
pl.sql()/SQLContext), stricter schema validation (collect_schema()). Verdict: "hardening the core," closing a long-standing stability gap with DuckDB rather than adding new capability. - DuckDB 2.0 release highlights: the "Quack protocol" (network access to a DuckDB instance — DuckDB-as-a-server), better async I/O for Parquet/CSV over S3, a first-class VARIANT type (semi-structured data), triggers, an extensible SQL parser, and "agent mode" (named in the release list but never explained or exercised in the piece — a real gap in an otherwise hands-on review). Beach reads the Quack-protocol/server direction as DuckDB chasing a bigger audience post-AWS-acquisition, not staying a purely embedded tool.
- Benchmark: 23GB of Backblaze hard-drive-failure CSVs on S3, windowed LAG + group-by aggregation, run on a deliberately undersized Linode box (2GB → 4GB → 8GB for Polars).
- DuckDB 2.0: succeeds on the first try at 2GB, 837.32s, visibly spilling to disk.
- Polars 2.0: silently dies (no error, no output) at 2GB, 4GB, and 8GB. Root cause per the release notes: spill-to-disk in 2.0 covers sort/window/"many expressions" but explicitly not group-by or join yet.
- Polars 2.1 (released the following week, adding group-by/window spill): 438.47s on the same 2GB box — 2x faster than DuckDB 2.0.
- Closing frame, consistent with Beach's other DEC pieces: read release notes for direction, not just feature lists, and keep testing hands-on rather than trusting vendor benchmarks or AI-written takes.
⚠️ Sponsorship
sponsored: true, sponsor_entity: LakeSentry. Explicit mid-issue block ("today's sponsor, LakeSentry!") pitching a Databricks/cloud cost-observability tool (cost allocation by team/center, anomaly detection on idle warehouses and orphaned clusters, MCP-based cost exploration) — unrelated to the Polars/DuckDB content, a pure ad placement rather than a topic-driven partnership. New sponsor, not previously logged in this project's DEC tracking — extends the now-confirmed unsettled/rotating posture: Delta Lake (standing through 2026-09-14) → Rudderstack (10-14) → PhoenixAI/StarRocks (10-19) → GizmoData (10-21) → none (10-26) → LakeSentry (11-02), a fifth distinct sponsor in six tracked issues with one no-sponsor gap in between. No bias touches the DuckDB/Polars verdict itself — LakeSentry has no stake in either engine.
Mapping against Ray Data Co
Concrete connection: RDCO's own knowledge-graph store (~/.claude/state/graph.duckdb, built and queried by the /graph-ingest.py + /graph-query tooling this very skill run depends on) is already built on DuckDB. It's a small file today (~12MB), nowhere near this piece's 23GB/2GB-box stress case, but this issue's verdict — DuckDB 2.0 survives a 10x-undersized-memory aggregation cleanly via spill-to-disk, while Polars 2.0 fails the identical job outright — is a stability data point in favor of that existing, unexamined infra choice holding up if the graph ever grows faster than expected. Not a reason to switch anything; a reason the choice already made has margin.
Second connection, stronger as a thread than as new fact: this is the third DEC issue the vault tracks on the exact DuckDB-vs-Polars stability axis — [[2026-04-09-data-engineering-central-replacing-polars-with-duckdb]] (Beach's original production migration off Polars), and [[2026-10-05-data-engineering-central-duckdb-polars-spark-serverless-benchmark]] (same Backblaze dataset, Databricks/Unity-Catalog environment, Polars OOM'd twice there too). This issue strips out the Databricks/Unity-Catalog confound from Oct 5 (plain S3 this time) and gets a mechanistic answer instead of a repeated anecdote: Polars' OOM problem on this class of job wasn't a vague "stability issue," it was a documented, named gap (no group-by/join spill in 2.0) — and it closed in exactly one point release. That's a useful general pattern for RDCO's own tool-evaluation habits beyond data engineering: a hands-on failure that looks like a durable verdict can be a dated implementation gap instead, and checking the next release before encoding the belief matters.
RDCO's own thesis is agent-harness engineering, not lakehouse tooling, so this stays medium rather than strong — no decision or direction here, just a reinforced empirical thread plus one real (if small-scale) infra validation point.
Related
- [[2026-10-05-data-engineering-central-duckdb-polars-spark-serverless-benchmark]] — same author, same Backblaze dataset, the Databricks/Unity-Catalog-environment predecessor to this S3-only rerun
- [[2026-04-09-data-engineering-central-replacing-polars-with-duckdb]] — Beach's original production-stability verdict against Polars that this issue's root-cause finding (no group-by spill in 2.0) explains more precisely
- [[2026-06-22-data-engineering-central-datafusion-comet-spark]] — same author's recurring "read the release notes for direction, test it yourself" method applied to a different engine