06-reference

analytics engineering roundup catalog is all you need

2026-06-14·reference·source: Analytics Engineering Roundup·by Anders Swanson
apache-icebergdata-catalogdbtduckdbsnowflakedatabricksdata-engineeringinteroperability

A Catalog Is All You Need — Anders Swanson

Anders Swanson (dbt Labs) argues that the Iceberg REST Catalog (IRC) has crossed a threshold: the data catalog is now the universal unit of interoperability across query engines and data platforms, not warehouse-specific connectors or copy jobs.

Issue contents

The demo: A 4-model DuckDB dbt project that writes directly into Snowflake's "Horizon" catalog or Databricks' "Unity Catalog" by changing a single YAML line (catalog_name: horizon). No Spark, no JARs, no warehouse bill. Platform complexity is pushed into catalogs.yml (configured once by a platform team), leaving every analyst with a one-line config.

The thesis — "a catalog is a bookmark, not a technology": IRC decouples the catalog (where tables live) from the query engine that reads/writes them. The catalog becomes the abstraction; the underlying format (Iceberg, DuckLake, future formats) is a swappable implementation detail. Swanson frames Apache Iceberg as "the Antarctic Treaty" — neutral territory where vendors compete and cooperate simultaneously.

State of the ecosystem: Both Snowflake and Databricks now have GA support for external engine writes to their proprietary catalogs (Horizon and Unity Catalog respectively). Databricks also announced OpenSharing last week. Swanson calls out the DuckDB Iceberg extension (v1.5.3, soon to support Iceberg v3) as reaching stable territory.

What's next: DuckLake as a forcing function to improve Iceberg v4's "many files" problem; format diversity absorbed by the catalog abstraction layer. Prediction: DuckLake will be remembered as a competitive pressure on Iceberg, not a replacement.

Homework suggested: Connect PyIceberg or DuckDB to your platform's IRC; point local DuckDB at production Iceberg catalog for dev/CI without a warehouse bill; hand data to another team by writing directly into their catalog (cross-platform data mesh without FTP or bucket grants).

Technical weeds: dbt-core#15239 enables DuckDB writes to Unity Catalog, Snowflake Horizon, and others. Both dbt and DuckDB currently encode per-IRC quirks; the end state is IRC standardization or capability advertisement via a capabilities/ REST endpoint.

Why this is in the vault

This is the clearest single-essay articulation of the catalog-as-interoperability-primitive thesis that has appeared in AER to date. Swanson provides a working, reproducible demo (not vaporware) showing cross-platform catalog federation with DuckDB + dbt in 2026. The essay closes the loop on his March 2025 prediction about GA writes to Snowflake/Databricks catalogs — both shipped. The "homework" section is directly actionable for any RDCO client running on Snowflake or Databricks who wants to lower warehouse bills or federate data to ML teams.

Mapping against Ray Data Co

Strong signal for phData DSA work. The cross-platform catalog play is immediately deployable in client engagements: if a client runs Snowflake + a Databricks ML team, writing directly from DuckDB into Unity Catalog via IRC eliminates the "grant my service account access to your bucket" negotiation. This is the kind of "boringly correct" infrastructure architecture Ray should be able to demo in a discovery call.

AI agent integration angle: Swanson links an "iceberg explorer agent skill" — an AI agent that can answer questions about data in an IRC. This maps directly to the MAC architecture and the data-management agent swarm work Tristan documented in May. Worth pulling the linked skill for evaluation.

DuckDB as thin client: The demo normalizes DuckDB as a zero-cost dev/CI query engine pointed at production catalog — a pattern worth recommending to clients who want to reduce warehouse spend on dev queries.

Iceberg v4 / DuckLake: The competition between DuckLake and Iceberg v4 will resolve in the next 12-18 months. The catalog abstraction means clients don't need to bet on the format winner now — they should bet on the IRC abstraction.

⚠️ Sponsorship

None detected. This is authored directly by Anders Swanson of dbt Labs. dbt Labs is the company behind dbt — the demo uses dbt — but there is no explicit sponsor block and the entire piece is technical content, not a product pitch. The dbt/DuckDB collaboration mentioned is open-source work. Treat as independent editorial with inherent author affiliation to dbt Labs.

Related