06-reference

data engineering central adbc arrow database connectivity

2026-07-20·reference·source: Data Engineering Central·by Anonymous Rust Dev

"ADBC?" — Arrow Database Connectivity (Anonymous Rust Dev)

⚠️ Sponsorship

Delta Lake is the named sponsor of this issue. The author explicitly discloses the sponsorship and states they use Delta Lake daily. Treat performance claims about Delta-adjacent tooling with awareness of this relationship.

Why this is in the vault

A Rust developer's practical walkthrough of ADBC — Apache Arrow's answer to JDBC/ODBC — with working code examples in Rust and Python, explaining why the zero-copy columnar interface matters for analytical workloads.

The core argument

ADBC (Arrow Database Connectivity) is a client API specification introduced by Apache in early 2023. It does one thing: standardize how applications query databases and retrieve results — but the results come back as Arrow columnar RecordBatches instead of row-by-row JDBC/ODBC style.

The argument against JDBC/ODBC for analytical use:

ADBC sidesteps both by treating Arrow as the native wire format. The driver doesn't define SQL dialect — it just passes your query string through and returns the result as Arrow data. Under the hood, for row-based databases like PostgreSQL, the driver uses the COPY binary protocol and converts to columnar in one efficient pass.

Stable ADBC driver targets (PyPI packages available):

Rust quickstart:

use adbc_core::{Connection, Database, Driver, Statement};
let mut driver = adbc_datafusion::DataFusionDriver {};
let db = driver.new_database().unwrap();
let mut conn = db.new_connection().unwrap();
let mut stmt = conn.new_statement().unwrap();
stmt.set_sql_query("SELECT 1").unwrap();
let reader = stmt.execute().unwrap();

Python quickstart (DuckDB):

import adbc_driver_duckdb.dbapi, pyarrow
data = pyarrow.record_batch([[1, 2, 3], ["a", "b", "c"]], names=["ints", "strs"])
with adbc_driver_duckdb.dbapi.connect("test.db") as conn, conn.cursor() as cur:
    cur.adbc_ingest("my_table", data)

The author notes ADBC's architecture mirrors OLTP patterns (ADO, JDBC): establish connection → prepare query → execute → enumerate results over a reader interface. The selling points are zero-copy data movement, a consistent cross-language API, and Apache Foundation stewardship. The article references a recent Daniel Beach interview with Wes McKinney (Arrow's principal author) as context.

Mapping against Ray Data Co

The most concrete link is phData's Snowflake work: Snowflake has a stable ADBC driver, and if phData clients build Python data pipelines consuming Snowflake data into Arrow-native tools (Polars, DuckDB, PyArrow), swapping the Snowflake connector for the ADBC driver eliminates an extra serialization round-trip. Ray's DSA role is discovery/scoping/handoff rather than low-level driver implementation, so this is background knowledge rather than an immediate action — but it's the kind of architectural nuance that surfaces in client scoping conversations about pipeline performance.

Secondary link: RDCO's own graph.duckdb knowledge graph uses DuckDB, which has a fully implemented ADBC driver. If the graph query layer ever needs programmatic ingestion of Arrow data from external sources, ADBC is the right interface rather than building a custom DuckDB ingest path.

The CAF project (governed knowledge graph at phData) could also benefit here: agents querying databases via Arrow-native interfaces avoid the row-to-column serialization penalty on every retrieval, which matters at scale.

Mapping is medium — directional relevance to phData Snowflake pipelines and RDCO's DuckDB stack, not an immediate implementation task.

Related