06-reference

data engineering central databricks summit 2026

2026-06-19·reference·source: Data Engineering Central·by Data Engineering Central
databricksdata-architecturelakehousereal-time-analyticsOLTP-OLAP

"Review of Databricks Data + AI Summit 2026" — @DataEngineeringCentral

Why this is in the vault

Databricks' two flagship Summit announcements — Lakehouse//RT and LTAP — represent the most architecturally significant attempt yet to collapse the OLTP/OLAP/serving-layer split that defines most enterprise data stacks today, and both have direct implications for how RDCO designs client pipelines on Databricks.

The core argument

The author zeroes in on two announcements from the 2026 Summit while deliberately setting aside the AI agent feature flood (Genie Ontology, ZeroOps, Omnigent, etc.), calling those a "ho-hum" category where no clear winners exist yet.

Lakehouse//RT (powered by Reyden)

Databricks is introducing a new real-time compute engine called Reyden designed to serve millisecond-latency queries directly against Delta Lake — eliminating the dedicated serving tier (ClickHouse, Pinot, Druid, Redis) that most teams run alongside their lakehouse today. Claimed benchmarks: 16x faster than existing real-time layers, 10ms query latency, 12,000 QPS at low latency. The architectural collapse looks like this:

Before: Delta Lake → ETL/CDC → ClickHouse/Pinot/Redis → Apps
After: Delta Lake → Lakehouse//RT → Apps + Dashboards + AI Agents

Unity Catalog governance stays intact — security policies and lineage are defined once, not replicated across serving systems. The author calls it "one of the most strategically important announcements" but reserves judgment on whether benchmark numbers translate to real production diversity.

LTAP (Lake Transactional/Analytical Processing)

More architecturally radical: Databricks wants OLTP and OLAP to share a single copy of data. Instead of the 40-year assumption that apps live in Postgres and analytics live in a warehouse connected by CDC/Kafka/Airflow, LTAP proposes:

          Delta / Iceberg
                │
   ┌────────────┼────────────┐
   │            │            │
Lakebase    Lakehouse   Lakehouse//RT
  OLTP        OLAP       Real Time

Key distinction vs prior HTAP attempts: Databricks is NOT building one engine to do everything. They're unifying the storage layer while keeping engines specialized. Lakebase handles Postgres transactions; the Lakehouse handles analytics; Lakehouse//RT handles serving — all reading the same governed Delta/Iceberg data.

The author surfaces a telling data point from Ali Ghodsi: roughly 80% of databases on Lakebase are already being created by AI agents, not humans. His framing — that the pressure to remove data duplication is now driven by agent throughput, not developer velocity — is the sharpest signal in the piece.

Mapping against Ray Data Co

Direct relevance to phData (main bet): If clients are on Databricks, LTAP and Lakehouse//RT materially change the discovery/scoping conversation. The old recommendation to "add ClickHouse for real-time" or "set up CDC from Postgres into the lake" may become unnecessary within 12-18 months. As a DSA at phData, surfacing this architectural shift in discovery calls is a differentiated positioning move — most SEs won't know it yet.

AI agent infrastructure angle: Ghodsi's framing that agents create databases 80% of the time on Lakebase maps directly to RDCO's own work. The Claude Code COO agent already creates and queries data structures programmatically. As RDCO builds more agentic pipelines for clients, "agent-native infrastructure" (single copy, no ETL, low-latency retrieval) becomes a selection criterion worth tracking. Lakehouse//RT's explicit callout of "AI agent retrieval" as a target workload validates this.

Gap to watch: The author flags what Databricks is NOT showing yet — whether Lakebase can handle production OLTP at massive scale, and whether LTAP survives "messy enterprise workloads." These are exactly the friction points that will come up in phData client conversations. File a follow-up note when preview details emerge (6-month horizon).

Confirmed skip signals for now: Genie Ontology, ZeroOps, Omnigent, Unity AI Gateway — author categorizes these as "everyone is throwing stuff at the wall." RDCO's [[llm-ops-patterns]] assessment should remain vendor-agnostic until a clear winner emerges from the Databricks AI feature set.

Related