Data Engineering Weekly #296
Why this is in the vault
Pinterest's loop-engineering fix for an analytics agent that skips steps and GoDaddy's template-driven pipeline agent are two more production data points for agent-compliance testing and "existing standards as agent starting point", both patterns RDCO already uses.
The core argument
The editor's note says ontology and knowledge graphs are not new but became prominent once agents arrived, because business scale and data volume are an order of magnitude beyond a decade ago. He respects the "told you so" camp but asks how usable RDF, OWL2, SHACL and R2RML really are in today's business setting. He also reads PostHog's post as the end of the modern data stack era, with catalogs turning into context engines.
Curation section
- PostHog (X post) - the modern data stack "had a good run"; teams are being bundled whether planned or not.
- GoDaddy - AI agent that asks for missing details, reads source data, and generates pipeline files plus a PR; existing templates and naming standards give the agent its starting point.
- Sponsored - StarTree: report on five open-table (Iceberg/Delta) trends for 2027, including agents creating new query patterns.
- Uber - moved much of Presto's execution to Velox (batch-oriented engine), validating against the old system and closing compatibility gaps; faster engines still need testing and ops support.
- HuggingFace / Dharma AI - small task-specific model that turns document images into structured JSON fields; worth testing against general-purpose models for high-volume extraction.
- Sponsored - Astronomer: 100+ page guide to AI orchestration with Apache Airflow (LLM, agent, multi-agent patterns, guardrails, human review).
- Rippling - fixed Flink data skew by changing record distribution and write batching; reported 27% faster and about $1M annual savings.
- Pinterest - analytics agent skipped steps like checking table details; built a test loop that proposes changes, checks when a step is needed, and rejects regressions. Lesson: test when an agent should act and when it should skip.
- Meta - "Intelligent Delivery" routes extra compute to where it helps ads models most.
- Anton Zhiyanov - probabilistic data structures in Go; foundational for scale.
Deep-fetches: 0 (curation issue; all items read from the newsletter's own blurbs, which were specific enough).
Mapping against Ray Data Co
Pinterest's analytics-agent compliance loop is the concrete connection: it tests both required and unnecessary steps and rejects changes that break other behavior, which is the same discipline as RDCO's fresh-eyes critic gates and behavior contracts, and a sharper version of the loop-engineering idea already filed in the vault. GoDaddy's agent, which bootstraps from existing templates and naming standards, echoes the station-based brigade (spec to tests to code) and the DEmate lesson from #295. The editor's ontology skepticism is a useful caution for any RDCO semantic-layer or knowledge-graph work: standards matter less than whether they are usable by agents.
⚠️ Sponsorship
Two paid placements: StarTree (open-table trends report) and Astronomer (Airflow AI orchestration ebook). Both are vendor promotion, not independent analysis; Astronomer has now sponsored at least three consecutive issues (#294-#296), and StarTree appeared in #295 too.
Related
- [[2026-11-09-data-engineering-weekly-295-demate-agent-coding-standards]]
- [[2026-11-02-data-engineering-weekly-294-agent-governance-controls]]
- [[2026-07-04-addy-osmani-loop-engineering]]
- [[2026-08-14-data-engineering-weekly-agent-ontology]]
- [[feedback_fresh_eyes_subagent_for_own_artifacts]]