SC 011 — Hi Again, Forgiving Failure, & Loops in DAGs

Summary

The newsletter relaunch. Three major life changes: moving to consulting at Mammoth Growth, beginning to work in public (YouTube, social media), and welcoming a five-month-old daughter.

Test Forgiveness: Proposes a framework for handling data quality test failures. Instead of three bad options (allow errors through, quick patch, refuse responsibility), use dbt’s warn_after and error_after settings, set timeboxes on issues with where config, and document exceptions. Balances timeliness against correctness.

Human-in-the-DAG: Three escalating levels for incorporating business stakeholder input into data classification: (1) seed files with manual CSVs, (2) reading from Google Sheets, (3) bidirectional integration where pipelines both read inputs and write back flagged issues. Level 3 remains aspirational but tools like Sigma input tables and dbt-duckdb external materializations are making it possible.

Key Arguments

• Test Forgiveness balances timeliness vs. correctness — a pragmatic middle path
• warn_after and error_after are underutilized dbt features
• Timeboxing data quality issues prevents alert fatigue without ignoring problems
• Human-in-the-DAG should be bidirectional — the pipeline should talk back to stakeholders
• Working in public is a deliberate practice shift, not just content creation

Writing Style Notes

Relaunch energy — personal, vulnerable (new baby, career change, public commitment). The “Test Forgiveness” and “Human-in-the-DAG” concepts are original frameworks, not just commentary. This is the founder at his most inventive.

Connections

• 01-projects/newsletter/index — the Sanity Check relaunch
• 01-projects/squarely-puzzles/index — mentioned in passing