06-reference/concepts

emergent correlated failure

2026-09-27·reference·status: draft
harness-thesissystems-failurecorrelation-riskcandidate-promotion

Emergent Correlated Failure: When Individually-Correct Local Decisions Add Up to a Worse Global Outcome

The one-sentence claim

A system built from parts that are each locally correct can still fail globally, when those parts' outputs synchronize — either in space (many similar local optimizations converging on the same downstream point at once) or in time (unrelated shocks clustering inside one window and killing something none of them could have killed alone).

Why this is in the vault

This is distinct from [[layered-defense-architecture]], which is about defending against failure that's already anticipated. CA-018 is about failure modes that get created by well-meaning local optimization, and the vault has both a spatial and a temporal variant. [[2026-04-20-practical-engineering-do-retention-ponds-actually-work]] is the spatial canonical case: individual on-site detention basins are each sized to meet their own peak-discharge ordinance, and each one is locally compliant — but when many basins along the same watershed use similar outlet controls, their attenuated peaks can synchronize at a downstream confluence and produce a worse flood than if none of the basins existed at all. [[2026-04-20-practical-engineering-spillway-failed-on-purpose]] gives a second spatial instance: Asheville's redundant transmission line and the original line were routed to share a downstream channel, so the "redundancy" shared a single failure mode and Hurricane Helene took out both together. [[2026-04-20-practical-engineering-californias-tallest-bridge-has-nothing-underneath]] gives the temporal variant: the Auburn Dam project didn't die from one cause — the 1975 Oroville earthquake, the 1976 Teton Dam collapse, and an April 1976 geology report all landed within a single year, and it was the clustering of three independently-explicable shocks, not any one of them, that killed institutional momentum for good.

Mapping against Ray Data Co

The spatial variant is an audit question for skill and infrastructure design: do multiple skills share retry/backoff windows or a downstream resource in a way that looks redundant but actually means one upstream event takes all of them out together (the Asheville failure mode)? Regional coordination — a shared rate-limit budget, a single circuit breaker instead of N independent ones — is the fix, the same way the retention-pond fix is a regional detention facility rather than N separately-compliant local ones. The temporal variant is a triage discipline: when 3+ unrelated failures land inside one cron cycle or one review window, the right response is to treat the cluster itself as the signal (something about the operating environment changed) rather than investigating each failure as an independent incident and missing that they're arriving together. This is a genuinely different alarm than any single-failure alert would trigger, and nothing in the current cron/audit stack explicitly watches for temporal clustering — it's currently only caught if a human notices the pattern by eye.

Related

Confidence

Three sources, canon-tier promotion bar met, but the spatial and temporal variants only share one common source (the spillway video) — the retention-pond and Auburn-dam sources are otherwise independent illustrations of the same shape rather than one tight cluster. All three are still Practical Engineering. The temporal-clustering-as-signal claim for RDCO's own cron/audit stack is this write-up's own extrapolation and isn't yet backed by an incident where it would have caught something the current per-failure alerting missed — worth revisiting once (or if) that happens.