"Delta Kernel (with Rust). Doing stuff." — Daniel Beach, Data Engineering Central (Oct 19 2026)
Why this is in the vault
A hands-on build log for Delta Kernel — the reusable library that lets any query engine read/write Delta Lake tables without reimplementing the Delta protocol itself — which is exactly the kind of protocol-abstraction layer an agent-facing data stack needs so engines don't each carry their own brittle Delta-version-coupling logic.
The core argument
- Delta Kernel exists because the Delta Lake ecosystem (DuckDB, Polars, Unity Catalog, deletion vectors, etc.) moves faster than every downstream tool can track; Kernel is a reusable library so engines get correct Delta read/write without implementing the Delta protocol themselves. Two flavors ship: Rust and Java, aimed at the languages behind current and future tooling.
- Beach builds a toy CLI ("delta-import-export") using
delta_kernelin Rust to read and write a Delta table on S3, walking through the core primitives: anEngineabstraction for interacting with Delta, andSnapshot/Scan/TransactionAPIs for read vs. write paths. He flags the Rust crate as explicitly experimental, "especially write," and notes the intro blog post he used for reference was already stale — the API surface is moving fast (an earlier, simpler Table API has already been replaced). - Working example:
cargo add delta_kernel, build aSnapshot, open aScan, iterate the engine data; for writes, build aTransactionoff aSnapshot, setwith_operation("WRITE")/with_engine_info/with_data_change(true), get awrite_context, thencommit. He notes production code needs retry handling around the commit loop. - Caveat: this basic path doesn't handle multi-writer production Unity Catalog Lakehouses — there are separate Unity Catalog crates for that, which he defers to a future post.
- Closing argument: Delta Kernel/Unity crates are unglamorous backbone infrastructure that make tools like DuckDB's native Delta support possible; the engineers building that plumbing (Databricks/Delta Lake team) do more to open up the Lakehouse to the ecosystem than critics who complain about Databricks lock-in acknowledge.
⚠️ Sponsorship
sponsored: true, sponsor_entity: PhoenixAI (NEXT QUERY 2026 / StarRocks). Delta Lake had been this sender's default sponsor for 9+ issues through 2026-09-14, then broke at the 2026-10-14 issue (Rudderstack, opening a CDP-vendor series). This issue breaks the pattern again: no Delta Lake sponsor block, and the Rudderstack CDP series doesn't continue here either. Instead there's a distinct grey callout-block (visually set off from body copy, same placement convention as prior sponsor slots) promoting "NEXT QUERY 2026," a free virtual summit on Nov 4 "hosted by PhoenixAI, the creators of StarRocks," pitched twice in the issue with highlighted anchor text. Bias implication is minor for this piece specifically — the technical walkthrough itself (Delta Kernel/Rust) isn't StarRocks-adjacent content, so the sponsor doesn't shape the argument the way the Rudderstack CDP series did. But it confirms Data Engineering Central's standing-sponsor slot is now unsettled issue-to-issue (Delta Lake → Rudderstack → PhoenixAI/StarRocks in three consecutive tracked issues) rather than reverting to the long-run Delta Lake default.
Mapping against Ray Data Co
Direct reinforcement of RDCO's "explicit agentic boundaries" / contract-layer thesis from [[2026-06-24-data-engineering-central-semantic-layers-agents]]: Delta Kernel is a concrete instance of the pattern RDCO needs generally — a stable, versioned abstraction layer that absorbs protocol churn so the tools built on top (an agent, a query engine, a pipeline) don't each need bespoke version-coupling logic against a fast-moving storage protocol, the same control-plane shape the vault's [[semantic-layer]] concept tracks one layer up the stack (metrics/query contracts rather than storage-protocol contracts). The explicit "experimental, especially write" caveat on the Rust crate is a useful calibration data point: any RDCO build that would lean on Delta Kernel for writes today should treat it as early and validate commit/retry behavior directly rather than trusting the abstraction, which echoes the field-report caution in [[2026-08-10-data-engineering-central-quasi-agentic-pipelines-databricks-airflow]] about non-deterministic/unproven pieces needing hard scaffolding rather than blind trust. Also extends the same sender's [[2026-06-17-data-engineering-central-lakehouse-delta-lake-rust]] (delta-rs guest post) and [[2026-07-20-data-engineering-central-adbc-arrow-database-connectivity]] (ADBC) thread on Rust-native Lakehouse connectivity options — Kernel, delta-rs, and ADBC are three different abstraction layers solving overlapping "how do I correctly talk to Delta/Arrow without reimplementing the protocol" problems, worth keeping distinct if RDCO ever evaluates a Rust-based data-access layer.
Related
- [[2026-06-17-data-engineering-central-lakehouse-delta-lake-rust]] — same sender's guest post on delta-rs, the other major Rust-native Delta Lake abstraction, useful contrast to Delta Kernel's engine-abstraction approach
- [[2026-07-20-data-engineering-central-adbc-arrow-database-connectivity]] — same sender's prior piece on ADBC, another protocol-abstraction layer in the same Rust/Arrow/Lakehouse connectivity space
- [[2026-06-24-data-engineering-central-semantic-layers-agents]] — same sender's argument that agents need a stable semantic/contract layer, which Delta Kernel exemplifies at the storage-protocol level
- [[2026-08-10-data-engineering-central-quasi-agentic-pipelines-databricks-airflow]] — same author's field report on not over-trusting non-deterministic/unproven pipeline pieces, same caution this issue's "experimental, especially write" flag calls for