Analytics Engineering Roundup — I built a (very small) agent swarm

Why this is in the vault

Tristan Handy (dbt Labs founder/CEO) time-boxed an 8-hour experiment building a working multi-agent "data management agent swarm" against his own internal dbt project, and reports the patterns and failure modes empirically. Despite the curation-sender slug, this issue is a hands-on build writeup, not a link roundup. It is directly load-bearing for RDCO's pipeline-* scaling thesis: it is a real-world account of decomposing data work into per-task agents and the infrastructure they share, written by the person who effectively defined the modern analytics-engineering category. The agent-swarm framing is the exact pattern RDCO is betting on for scaling pipeline fan-out from single-digit to 20-50 concurrent jobs.

The core argument

The piece opens from a quote by Jordan Tigani (Motherduck) predicting the settled data form factor will be "an agent swarm for data management backed by a query engine." Handy is reflexively skeptical of the term "agent swarm" but resolves the skepticism by building rather than theorizing.

His decomposition: data management is a long list of small, individually-not-hard tasks (profile a column, document an object, debug a pipeline, etc.). The agent-swarm thesis is to give each task a purpose-built agent with the right skills and context, set them to continuously observe their domain, proactively detect problems, and proactively fix them — humans-in-the-loop at first, autonomous over time.

What he actually built, and what he learned:

• Shared context is the real foundation, not the agents. A big chunk of the 8 hours went to assembling a consistent context layer that nearly every agent needs regardless of its job: standard column profiling (descriptive stats), dbt metadata, and query history. (He flags git history and prod run logs as obvious future additions.)
• dbt as a single connection broker. Codex (GPT-5.5) built scaffolding that runs all warehouse queries through dbt show, eliminating the need for multiple connection pathways to the warehouse — one disciplined data access path for every agent.
• Fan-out worked mechanically. The profiler spun up parallel workers to execute the profiling queries and wrote results to a markdown context file. Wiring took effort but the build itself was straightforward and high-quality.
• The first useful agent compounds the next. Reading the context output, the obvious first task became filling in dbt description fields — and crucially, better descriptions improve every downstream agent because they all consume those descriptions as their own context. Quality begets quality.
• Naive description generation is useless; tool-using "research mode" is the unlock. The first-cut description agent just restated column names (a boolean is_duplicate_customer got "is this row a duplicate"). The fix: give the agent the associated dbt code as context AND the ability to get curious — propose follow-up questions, write safe read-only SQL, run it via dbt show, store the evidence, then synthesize. The improved output correctly explained that duplicates arise from concurrent Stripe + Metronome accounts sharing a customer_id and that the model filters to keep the non-Metronome record.

His closing claim: the bottleneck on data-system quality was never capability, it was contention for skilled humans' time. Decompose the work task-by-task, eliminate that constraint per-task, and existing systems "skyrocket in quality." He believes there are no hard CS problems left here, only experimentation plus domain expertise, and exited the experiment more bullish than he entered. He plans to deploy it internally.

Mapping against Ray Data Co

This is the most concrete external validation yet of the pipeline-* fan-out thesis, from a credible builder, with transferable primitives rather than hype. Concrete takeaways for scaling RDCO's pipeline seats from single-digit to 20-50 concurrent jobs:

1. Build the shared-context layer before scaling agent count. Handy's biggest time sink was the context substrate (profiling + metadata + query history), not the agents. RDCO's analog: the per-run scratch dir and spec/tests/code artifact chain in the 4-seat pipeline (pipeline-spec-author → pipeline-test-author → pipeline-code-author → pipeline-critic) is already a shared-context pattern. The lesson is to invest in the substrate first; fan-out is cheap once context is solid. This is the same "master business-context folder" insight already in the vault.
2. A single connection broker scales better than N pathways. dbt-as-broker (dbt show) means every agent shares one disciplined data-access path. RDCO analog: a single canonical tool/wrapper per external resource (the 1Password wrapper-script pattern, the cloudflare-api.sh wrapper) rather than each agent reinventing access. Worth auditing whether pipeline seats route through shared brokers vs ad-hoc connections as the count grows.
3. Compounding-context ordering: do the agent that improves other agents first. The description agent was valuable mostly because its output feeds every downstream agent. For RDCO this argues for sequencing fan-out so foundational/context-producing seats run before consumer seats — and is a direct argument for self-evolving skills (see SkillOpt note): the act of doing the work should improve the substrate the next run inherits.
4. Tool-using "research mode" is the quality line, not raw model capability. The difference between useless and genuinely-useful output was giving the agent code-context + the ability to ask its own questions, write read-only SQL, execute, and synthesize. This is the IC-mode-vs-production-mode distinction in RDCO terms: a single-shot agent produces slop; an agent that can investigate and gather evidence produces value. Reinforces the "no slop cannon" rule and the verification-as-independent-worker pattern.
5. Time-box and ship-internal-first is a viable validation cadence. 8 hours of calendar time to a deployable-internally proof of concept is the kind of small-bet discipline that fits RDCO's "bets are downstream of agent capability" stance — build the capability, prove it on internal surfaces, then scale.

Caveat for calibration: Handy is constitutionally an AI optimist (he says so) and this is a single n=1 experiment on his own polished internal stack with infra already in place. The "no truly hard CS problems left" claim is the optimist's framing; coordination of many always-on agents (the actual "swarm" part) was explicitly NOT built — he built sequential single-task agents, not a coordinated swarm. The hard part (orchestration, conflict resolution, autonomous-fix safety) remains unproven. Treat as strong directional evidence for the per-task-decomposition + shared-context primitives, weaker evidence for autonomous-swarm coordination.

Mapping strength: strong on the decomposition/shared-context/connection-broker primitives; medium on the autonomous-coordination thesis (asserted, not demonstrated).

Sponsor / bias note

No sponsor block, no tracked third-party promo. The only outbound links are self-cross-promo within the dbt/Substack ecosystem: a reference to an upcoming podcast episode with Jordan Tigani (Motherduck) and a Substack redirect to one of Tigani's posts. Author bias is disclosed in-text (self-described AI optimist). This is dbt Labs' founder writing about a build on dbt's own tooling (dbt show, dbt metadata, description fields) — implicit promotion of the dbt-as-control-plane position, though framed as an honest experiment rather than a pitch.

Related

• [[2026-05-03-alphasignal-single-vs-multi-agent-systems]] — the single-vs-multi-agent architecture tradeoff this build sits inside
• [[2026-05-26-skillopt-self-evolving-agent-skills]] — self-evolving skills as the mechanism for the "compounding context" insight (each run improves the substrate the next inherits)
• [[2026-01-11-ae-roundup-ai-agents-data-lake]] — earlier same-source treatment of agents operating directly on the data stack
• [[2026-05-20-writewithai-master-business-context-folder]] — the shared-context-substrate-before-agents lesson, restated