Benchmarking Vortex File Format vs Parquet, CSV, Lance (DuckDB / Polars / Datafusion)

Why this is in the vault

Beach does a first-poke hands-on benchmark of Vortex, a new columnar file format positioning itself as a Parquet successor (100x faster random access, 10-20x faster scans, 5x faster writes, similar compression). Vortex is now an LFAI&Data incubation-stage project (vendor-neutral governance, originally out of spiraldb), format stable as of v0.36 though library APIs still churn. RDCO's data-engineering surface lives downstream of exactly these choices — file format and query engine — so a credibility check on a "Parquet killer" before it ever shows up in a client pipeline is worth filing. Headline takeaway is a negative one: the format may be real, but the Python integration layer is not yet production-grade.

The core argument

Beach runs the same aggregation query (Backblaze hard-drive failure dataset, ~24 GB) across CSV → Parquet → Vortex, using DuckDB, Polars, and Datafusion as the engines, to see whether Vortex's hype survives contact with real Python tooling.

What won / lost:

• CSV baseline was slow (DuckDB ~25s); Datafusion was the standout on raw CSV (~5s) while Polars OOM'd — which Beach uses to relitigate his standing grievance that Polars is unreliable under memory pressure (he says he pulled it from production over this).
• Parquet collapsed runtimes to sub-second across all three engines (0.12–0.37s) — the real "wow" jump in the piece is CSV→Parquet, not anything→Vortex.
• Vortex matched Parquet on speed (pure Vortex scan with filter pushdown ~0.11s, Polars LazyFrame ~0.11s, DuckDB via PyArrow ~0.20s) but did NOT beat it meaningfully at this dataset size.

The actual takeaway: the integrations "aren't as solid as they claim." The DuckDB Vortex extension OOM'd on directories of files, Polars wouldn't accept glob patterns and forced an immediate convert-to-Arrow/PyArrow detour, and the working code was ugly. Beach's verdict: the format itself may shine on much larger datasets, but right now it isn't worth the hassle versus mature Parquet tooling. He flags this as honest early-lifecycle friction, not a fault of the third-party engines. One caveat on rigor — this is a single mid-size dataset on one machine, explicitly a casual holiday-weekend poke, not a controlled benchmark.

Mapping against Ray Data Co

Medium relevance, and the value is mostly as a "not yet" signal rather than an adopt signal.

• Client-reporting / data-engineering substrate: Parquet remains the correct default. The load-bearing lesson is the CSV→Parquet jump (200x here), which is the move that actually matters for any RDCO pipeline still touching raw CSV. Vortex offers no demonstrated marginal win over Parquet at the scale RDCO operates, and carries integration risk. Skip Vortex for now; revisit if a client workload is genuinely random-access-heavy or object-storage-bound at large scale, where Vortex's claimed 100x random-access edge could matter.
• D1 / health pipeline: Not relevant. D1 is SQLite-shaped row storage; columnar file-format choice doesn't enter that path. The personal health-data volumes are nowhere near the regime where any of this moves the needle.
• Tooling-decision discipline: The more durable RDCO takeaway is methodological — the gap between a format's marketing-page benchmark claims and its real Python integration maturity. "Stable format spec, churning library APIs" is exactly the failure mode to probe before adopting any shiny tool into a billable pipeline. Consistent with prior DEC notes preferring DuckDB's reliability over Polars' unpredictability.

Related

• [[2026-04-09-data-engineering-central-replacing-polars-with-duckdb]] — same author's reliability case for DuckDB over Polars; this note's Polars OOM is a callback to it.
• [[2026-05-22-data-engineering-central-declarative-functional-composition]] — recent DEC thought-leadership in the same engine/tooling lane.
• [[2026-05-18-data-engineering-central-spark-postgres-duplicates]] — adjacent DEC hands-on data-engineering piece.

Source fidelity: full plaintext body rendered from Gmail thread. Vortex maturity/governance facts confirmed via one fetch of the project repo. No sponsor block present; standard Substack reader-supported CTA only. Benchmark is single-dataset, single-machine, self-described casual — treat speed figures as directional, not authoritative.