06-reference

data engineering weekly 290 agentic decision routing security

2026-10-05·reference·source: Data Engineering Weekly·by Ananth Packkildurai
data-engineeringjevllm-routingagentic-securitydecision-modelsairflowcuration

Data Engineering Weekly #290

Why this is in the vault

Issue 290 clusters three items around the same question — when should a decision be handed to a model versus a deterministic gate, and what stops an agent from making an unsafe one — plus an Airflow durable-execution update that bears on how RDCO's own Workflow scripts should checkpoint; the Jev-classifier thread from issue 289 continues via a second, independent author.

Mapping against Ray Data Co

The Sreedeepsinha piece on ai_decide vs ai_classify is close to a direct description of RDCO's own auto-mode classifier hard-gate (feedback_automode_classifier_hard_gate): the author's argument — ask the model multiple fixed questions with probabilities in one call, log the decision for auditability, and default uncertain or failed checks to the safer path — is exactly the shape of the deploy/production-write hard-stop RDCO already runs, just generalized past the one gate. Worth re-reading if the classifier-hard-gate design gets revisited, since this is independent confirmation the pattern is converging elsewhere, not a novel idea to adopt.

Two more connections, lower stakes: Josh Rosen's "Agentic Security Stack" (restricting agent tool/file access, validating instructions, independent reviewers plus automated testing to catch what review can't) parallels the posture behind RDCO's feedback_mcp_install_security_review_default and the verify-dispatch critic — same idea, applied to installing tools rather than running them. And Astronomer's Airflow 3.3 durable-execution update (checkpoint progress so retries reconnect to running jobs instead of repeating completed AI calls) is a concrete argument for why any new workflow-authoring script that calls an LLM mid-run needs resume logic, not just retry logic — a crash shouldn't mean re-billing a step that already finished.

Curation section

Top-of-issue paid ad (see Sponsorship): "How to Build a Data Platform From Scratch" eBook download — same lead-gen slot that ran in issue 289.

⚠️ Sponsorship

Two paid slots, neither with a vendor name in the plain text (Substack redirect links; branding is image-only): (1) the top-of-issue "How to Build a Data Platform From Scratch" / Data Platform Fundamentals eBook ad, and (2) "Sponsored: AI Modernization Guide," a legacy-pipeline-modernization download. Both are recurring — identical to two of the three paid slots in issue 289 — and both are lead-gen downloads unrelated to the curated items, so bias risk to item selection stays low. Separately, every curated item is first-party content: Uber, Gojek, Airbnb, WHOOP, and Astronomer are company/vendor engineering blogs with an implicit self-promotion angle (Astronomer in particular is selling an Airflow feature), and Raschka, Sreedeepsinha, Bourous, and Rosen are independent authors with no disclosed commercial tie to the newsletter.

Related