/process-newsletter Output Invariants

Why this exists

Kingsbury’s “Future of Everything Is Lies” critique landed a real punch: if the verification layer is itself an LLM, you’ve just stacked another correlated failure mode on top of the one you were trying to catch. Two LLMs that share training distributions, reinforcement signals, and prompt-injection susceptibility don’t independently verify each other — they fail together, quietly, and in ways the user can’t audit. RDCO took that critique seriously. This audit script is our concrete answer: a pure-Python deterministic test suite for /process-newsletter outputs that makes ZERO LLM calls and cannot hallucinate a pass. It can fail in known mechanical ways (regex misses an unusual heading variant, an edge-case YAML that parses but doesn’t match intent), but those failures are inspectable, reproducible, and patchable. The audit doesn’t replace LLM judgment — it provides a structural floor underneath it. If an LLM-generated newsletter note doesn’t even have its required frontmatter fields or the load-bearing “Mapping against Ray Data Co” section, no amount of LLM-on-LLM review will save it; the structural check catches it cheaply, deterministically, and forever.

The 13 invariants

How to extend

When adding a new invariant, follow this pattern:

1. Single concern. One invariant = one structural property. If you find yourself writing or between two unrelated checks, split them.
2. Deterministic. No LLM calls. No network. No probabilistic logic. Stdlib + PyYAML only. If a check requires judgment, it doesn’t belong here — it belongs in an LLM-driven review skill.
3. Named I<N>. Sequential, never reused. If you retire an invariant, leave its ID dead and don’t recycle it.
4. Document in this SOP. Add a row to the invariants table above with the same name as the script’s INVARIANTS dict entry.
5. Add to the script. Implement in ~/.claude/scripts/audit-newsletter-outputs.py, return (invariant_id, message) tuples on failure, empty on pass.
6. Dogfood-test against the existing corpus before committing. Run with --since 2026-01-01 (or the full audit window) and inspect the violation distribution. If the new invariant generates >50% violation rate against existing files, either the invariant is wrong or the corpus has a real hygiene problem worth surfacing — be honest about which.

Failure handling

The audit is a post-condition check, not a gating check. When /process-newsletter finishes a batch or watch run, it runs the audit, includes the violation summary in the run report, and surfaces failing files for founder review. It does NOT auto-block, auto-retry, or auto-fix. The pattern is discover → audit → decide, not gate → block → error. The founder owns the decision of whether a violation matters enough to re-process the offending file. This keeps the audit cheap and honest: it never gets in the way, and it never silently “corrects” something it doesn’t understand. If the audit becomes a gating check in the future (e.g. wired into a pre-commit hook on the vault repo), the --strict flag exists for that — but the default mode is observe-and-report.

Limitations (be honest)

These invariants do NOT catch:

• Semantic correctness of the Mapping section. I8 fires if the heading is missing. It cannot tell you whether the mapping argument is good, honest, or load-bearing. A note with ## Mapping against Ray Data Co\n\nN/A passes I8.
• Accuracy of sponsor disclosure. I11 fires if the ## Sponsorship heading is missing when sponsored: true. It cannot verify that the sponsor was correctly identified, that an undisclosed sponsor wasn’t missed, or that the bias note is calibrated.
• Whether the citation is actually right. Nothing here checks source_url resolves, that the author byline matches the real author, or that quoted material is accurate.
• Whether the topic-slug is well-chosen. I10 only checks the sender prefix. The trailing topic-slug could be lazy, generic, or misleading.
• Whether the article should have been filed at all. Skip-triage (Step 2.5 of the skill) is judgment territory; we can’t catch a file that shouldn’t exist.
• Cross-document consistency. If two notes contradict each other, or if a note contradicts something already in the vault, the audit doesn’t notice.

These remain LLM-judgment territory. Future versions could add semantic invariants that cross-reference against the QMD index or the typed knowledge graph at ~/.claude/state/graph.duckdb (e.g. “does this note’s claimed author exist in the vault’s author authority list?”), but that re-introduces LLM contamination if the index/graph were themselves built by LLM-driven processes. Keep that semantic layer separate, name it differently (e.g. semantic-audit-newsletter-outputs.py), and never confuse the deterministic structural audit with the probabilistic semantic audit. The whole value of this script is that it doesn’t pretend to do something it can’t.