06-reference

data engineering central polars 2 0 duckdb 2 0

2026-11-02·reference·source: Data Engineering Central·by Daniel Beach
duckdbpolarsduckdb-2.0polars-2.0benchmarkspill-to-diskstreamingoombackblaze-datasets3

"Polars 2.0 vs DuckDB 2.0" — Daniel Beach (Nov 2 2026)

Why this is in the vault

Beach re-runs his signature stress-test (undersized compute, oversized data, same Backblaze hard-drive-failure CSV dataset he used in the Oct 5 Databricks/Unity-Catalog benchmark) against the fresh 2.0 releases of both tools, and gets a sharper, more mechanistic answer than before: DuckDB 2.0 finishes a 23GB aggregation on a 2GB box in 14 minutes via its new spill-to-disk/async S3 I/O, while Polars 2.0 OOMs on the identical job at 2GB, 4GB, and 8GB — because Polars 2.0's advertised "default spill-to-disk" explicitly excludes group-by and window operations (confirmed by quoting Polars' own release notes). A coda one week later: Polars 2.1 adds group-by/window spill and then beats DuckDB 2.0 by 2x on the same job — the fastest a dataframe-tool stability gap has closed in this tracked thread.

The core argument

⚠️ Sponsorship

sponsored: true, sponsor_entity: LakeSentry. Explicit mid-issue block ("today's sponsor, LakeSentry!") pitching a Databricks/cloud cost-observability tool (cost allocation by team/center, anomaly detection on idle warehouses and orphaned clusters, MCP-based cost exploration) — unrelated to the Polars/DuckDB content, a pure ad placement rather than a topic-driven partnership. New sponsor, not previously logged in this project's DEC tracking — extends the now-confirmed unsettled/rotating posture: Delta Lake (standing through 2026-09-14) → Rudderstack (10-14) → PhoenixAI/StarRocks (10-19) → GizmoData (10-21) → none (10-26) → LakeSentry (11-02), a fifth distinct sponsor in six tracked issues with one no-sponsor gap in between. No bias touches the DuckDB/Polars verdict itself — LakeSentry has no stake in either engine.

Mapping against Ray Data Co

Concrete connection: RDCO's own knowledge-graph store (~/.claude/state/graph.duckdb, built and queried by the /graph-ingest.py + /graph-query tooling this very skill run depends on) is already built on DuckDB. It's a small file today (~12MB), nowhere near this piece's 23GB/2GB-box stress case, but this issue's verdict — DuckDB 2.0 survives a 10x-undersized-memory aggregation cleanly via spill-to-disk, while Polars 2.0 fails the identical job outright — is a stability data point in favor of that existing, unexamined infra choice holding up if the graph ever grows faster than expected. Not a reason to switch anything; a reason the choice already made has margin.

Second connection, stronger as a thread than as new fact: this is the third DEC issue the vault tracks on the exact DuckDB-vs-Polars stability axis — [[2026-04-09-data-engineering-central-replacing-polars-with-duckdb]] (Beach's original production migration off Polars), and [[2026-10-05-data-engineering-central-duckdb-polars-spark-serverless-benchmark]] (same Backblaze dataset, Databricks/Unity-Catalog environment, Polars OOM'd twice there too). This issue strips out the Databricks/Unity-Catalog confound from Oct 5 (plain S3 this time) and gets a mechanistic answer instead of a repeated anecdote: Polars' OOM problem on this class of job wasn't a vague "stability issue," it was a documented, named gap (no group-by/join spill in 2.0) — and it closed in exactly one point release. That's a useful general pattern for RDCO's own tool-evaluation habits beyond data engineering: a hands-on failure that looks like a durable verdict can be a dated implementation gap instead, and checking the next release before encoding the belief matters.

RDCO's own thesis is agent-harness engineering, not lakehouse tooling, so this stays medium rather than strong — no decision or direction here, just a reinforced empirical thread plus one real (if small-scale) infra validation point.

Related