06-reference

data engineering central quasi agentic pipelines databricks airflow

2026-08-10·reference·source: Data Engineering Central·by Daniel Beach
agentic-data-engineeringdatabricksapache-airflownon-determinismorchestration

"Quasi-Agentic Pipelines with Databricks and Apache Airflow" — Daniel Beach, Data Engineering Central

Why this is in the vault

A practitioner's field report on bolting LLM/agent calls onto an existing production Airflow + Databricks stack, rather than buying a new "agentic" platform — directly useful as ground truth for how deterministic data pipelines actually absorb non-deterministic steps.

The core argument

Beach argues most "Agentic + Data" content on LinkedIn is marketing noise, and that the real, unglamorous problem is bridging deterministic DAGs with non-deterministic LLM calls using tools teams already run. His concrete example: a production Airflow DAG that mixes normal Python tasks with API calls to a Databricks Model Serving Endpoint (he cites a real Terraform snippet spinning one up), with downstream tasks depending on outputs that "may or may not return what you expect." His core tenets are simplicity, designing explicitly for non-determinism, and using existing tooling until it actually breaks — not reaching for a new SaaS agent framework for every use case. He walks through Apache Airflow's new Common AI Provider (@task.llm, @task.agent with toolsets like SQLToolset, @task.llm_branch for LLM-decided routing, and LLMOperator with require_approval human-in-the-loop gates) as evidence the orchestration layer itself is absorbing this pattern rather than requiring a separate agent platform. He closes by reframing the real question as not "how do we insert LLMs into pipelines" but "how do deterministic pipelines integrate with non-deterministic LLM/agent steps" — a trust problem, since data trustworthiness is what businesses actually depend on.

Mapping against Ray Data Co

Directly validates the incremental, tools-you-already-have posture Ray Data Co takes toward agent integration (e.g. bolting Claude-driven steps into existing skill/cron infrastructure rather than adopting a dedicated agent-orchestration SaaS) — Beach's "core tenets" of simplicity and using existing tooling until it fails is the same instinct behind keeping the channels-agent stack on tmux + LaunchAgent rather than migrating to a heavier agent framework. It's also a useful counterweight on the CAF/Organizational Intelligence work: his "how do deterministic pipelines integrate with non-deterministic agent steps" framing is the trust-and-verification question RDCO's own workflow-agent output-integrity failure modes (pointer-not-content returns, false "verified" stamps) are instances of, just observed from the data-pipeline side instead of the agent-harness side.

⚠️ Sponsorship

Sponsored by Delta Lake — same sponsor as the 2026-06-11 "Escaping the Agentic Token Tax" issue. Beach discloses the sponsorship explicitly ("Thanks to Delta for sponsoring this newsletter... I use Delta Lake daily") and the sponsored product (Delta Lake) is not substantively discussed in this issue's technical content, which centers on Databricks Model Serving + Airflow instead — low direct bias risk on the actual argument, but Beach's self-disclosed daily-use relationship with the sponsor is worth flagging as a standing pattern across his DEC issues.

Related