"Apache Datafusion Comet (Spark Accelerator)" — @Daniel Beach
Why this is in the vault
A hands-on re-evaluation of Apache DataFusion Comet as a Spark native execution accelerator, with a sharp DX-failure thesis that maps directly to how phData client conversations about query performance actually stall out.
The core argument
Beach revisits Apache DataFusion Comet — an Arrow-native query execution plugin for Spark — after a poor 2024 test, asking whether it has improved. The technical answer is deferred behind a paywall, but the editorial frame is clear before it: tools fail mainstream adoption not because of speed deficits but because of poor Developer Experience. Comet is placed in the "technically impressive, DX-last" archetype, contrasted against DX-first tools like Databricks and MotherDuck/DuckDB.
Key diagnostic points from the free portion:
- Installation friction: The official GitHub/docs page lists supported Spark versions but does not link to pre-built JARs — forcing users to hunt Maven. Beach treats this as a first-class failure signal.
- Config surface: Comet ships with roughly 1,000 documented configuration parameters, with no "top 10" getting-started guide. The Compatibility section (detailing fallback behavior to vanilla Spark) is described as overwhelming rather than reassuring.
- Cluster setup (Databricks DBR 17.3 LTS / Spark 4.0): JAR delivered via Unity Catalog Volume + init script, four required Spark configs to activate the plugin and custom shuffle manager.
- Test dataset: Backblaze hard drive data, all of 2025, ~43 GB of CSVs. Framed as a realistic mid-size production pipeline, not a Big Data extreme.
- Paywall cut: Benchmark execution times and final verdict are behind the paid tier. The published free content ends at code setup.
The 2026 LinkedIn buzz around Comet is noted as the prompt for re-visiting it.
Mapping against Ray Data Co
phData client conversations: Any client running Spark on Databricks and asking "how do we make this faster without moving to Delta Live Tables or Photon?" is a natural Comet conversation. This article is useful context for setting realistic expectations — Comet's theoretical wins may be negated for most teams by the onboarding tax unless someone already owns the Spark config surface.
DX-as-adoption-barrier framing: The archetype contrast (DX-first vs. technically-impressive-but-inaccessible) is a recurring pattern in the data tooling market. Beach's framing is a clean vocabulary for positioning advice to clients evaluating open-source accelerators.
Deal Solutions Architect angle: When scoping a Spark optimization engagement, Comet is now a checkable hypothesis — worth a 2-hour spike to validate JAR compatibility with the client's DBR version before promising it as a lever. The ~1,000 config parameter warning is a scoping signal: budget config tuning time or rule it out.
Limitation: The article is paywalled at the benchmark results, so the actual performance delta since 2024 remains unconfirmed. Treat as "DX audit useful, perf verdict pending."
Related
- [[06-reference/apache-arrow-native-execution]] — DataFusion is Arrow-native; Comet inherits this lineage
- [[06-reference/databricks-runtime-compatibility]] — DBR version pinning is the first failure point for any Spark plugin
- [[02-sops/phdata-client-spark-optimization-checklist]] — candidate home for a Comet evaluation step
- [[06-reference/duckdb-motherduck-dx-benchmark]] — Beach's cited DX-first counterexample; useful contrast for client tooling conversations