"The Future of the Lakehouse: Delta Lake, Rust, and Data Platforms at Scale"

Podcast episode — Data Engineering Central Podcast, ~56 min Guest: Ethan Urbanski — maintainer of delta-rs, data engineer in the pharmaceutical industry Host: Daniel Beach

Why this is in the vault

This episode sits at the intersection of open table format trajectory (Delta Lake vs. Iceberg), the Rust-native data ecosystem (delta-rs moving Delta Lake outside Spark), and enterprise data platform design in regulated industries. Ethan's delta-rs maintainer perspective gives a practitioner-level view on where Delta Lake is heading that's harder to get from blog posts. The agentic analytics thread at the end connects directly to the RDCO AI-on-data thesis.

Key angles covered:

• delta-rs and post-Spark Delta Lake: Delta Lake has historically been Spark-native; delta-rs opens it to Python, Rust, and lightweight compute environments (DuckDB, Polars, etc.) without a JVM
• Open table format landscape: Practical Delta Lake vs. Apache Iceberg comparison from someone who maintains one of them
• Open catalogs: Catalog interoperability and what it means for vendor lock-in avoidance
• Governance and scalability in regulated environments: Big pharma data platform constraints (audit trails, validation, lineage) as a stress test for lakehouse architectures
• Rust in data tooling: Why Rust is appearing across the data stack (Arrow, DataFusion, delta-rs, Polars) and what it buys (performance, memory safety, portability)
• Agentic analytics and AI-enabled data systems: Near-future where AI agents query and reason over lakehouse data — emerging pattern for RDCO clients

Mapping against Ray Data Co

Strength: medium

• Lakehouse fluency: Ray's phData role will involve scoping and handing off data platform work. Understanding where Delta Lake is moving post-Spark (delta-rs, DuckDB integration) is directly relevant conversation material with data engineering practitioners
• Open format positioning: Clients will ask about Delta Lake vs. Iceberg. This episode gives a nuanced maintainer-level take vs. the vendor marketing layer
• Agentic analytics: The closing thread on AI-enabled data systems maps to the RDCO thesis — agentic workflows on top of lakehouse storage is the architecture pattern to track for client advisory and internal tooling
• Pharma regulated-environment angle: Compliance-heavy data platform constraints (validation, lineage, audit) appear in financial services and healthcare clients — patterns transfer across regulated verticals

Not an immediate action item. Files as foundational reference on open table format trajectory.

Related

• [[06-reference/2026-06-07-data-engineering-central-duckdb-unity-catalog-commits]] — same newsletter, DuckDB + Delta Lake + Unity Catalog catalog-commit pattern; directly adjacent to delta-rs usage outside Spark
• [[06-reference/2026-01-25-ae-roundup-iceberg-catalog-layer]] — Russell Spitzer on Apache Iceberg catalog layer and v1–v4 format evolution; counterpart to the Delta Lake perspective in this episode
• [[06-reference/2026-04-13-data-engineering-central-lambda-kappa]] — Daniel Beach on architectural foundations; Lambda/Kappa patterns that feed into lakehouse design decisions
• [[06-reference/2026-04-04-dedp-design-patterns-intro]] — DEDP open data platform / lakehouse pattern; theoretical grounding for what this episode demonstrates in practice