Why this is in the vault
Hands-on 2026 re-evaluation of Apache DataFusion Comet — a Rust-based Spark accelerator built on DataFusion — by an engineer who tried it in 2024 and found it painful. The article walks through the full setup path: finding the right Maven JAR for a given Spark/DBR version, required cluster configs, and the developer-experience friction points that still plague the project. The author's verdict is characteristically blunt: the tooling is clever but the developer experience is still a mess (1000+ config options, no prominent pre-built JAR links, a sprawling compatibility matrix). Strong signal that Comet is not yet mainstream-ready despite the performance claims.
Key technical details captured:
- Comet requires matching JAR to Spark version (Maven:
comet-common-spark4.0_2.13-0.16.0.jarfor Spark 4.0 / DBR 17.3 LTS) - Required Spark configs:
spark.plugins=org.apache.spark.CometPlugin,spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager, plus driver/executor classpath entries - Falls back silently to vanilla Spark where unsupported —
spark.comet.explainFallback.enabled=truesurfaces those fallbacks - Author's framing: developer experience, not raw speed, determines whether a tool goes mainstream (cites Databricks and DuckDB/MotherDuck as counterexamples done right)
Mapping against Ray Data Co
Medium. Ray's phData role involves Databricks-heavy client engagements. Comet is relevant as a client question ("should we use Comet to speed up our Spark jobs?") that may arise in DSA discovery conversations. The article provides enough grounding to give a credible, grounded answer: "it's promising but not production-ready without significant DX investment — unless you have a specific, well-understood workload where the fallback behavior is acceptable."
The developer-experience framing (positivity + DX-first > raw speed benchmarks) is also a useful mental model for evaluating any new data tooling a client is excited about. The Databricks/MotherDuck contrast is worth internalizing as a talking point.
Weaker relevance for RDCO product work — Comet is not a SaaS layer but a JVM plugin, so no direct skill-building angle.
Related
- [[06-reference/2026-06-22-data-engineering-central-datafusion-comet-spark]] — this note
- [[02-sops/phdata-role-dsa-context]] — DSA role context for client conversations
- [[06-reference/apache-datafusion-ecosystem]] — broader DataFusion ecosystem notes (if exists)
- [[06-reference/databricks-best-practices]] — Databricks-adjacent reference notes