Nobody names this failure mode, but both halves of the control have names - and the half RDCO reached for first is the wrong one
The question
Verbatim: "Is there a documented detection pattern for the "correct-rule-wrong-default" permission-failure mode (defensible access-control default that silently breaks automation, discovered weeks later)?"
Open follow-up #2 from [[2026-10-24-coded-vs-prompted-agent-permission-gates]], which established that Every's July skill-ownership lockout and RDCO's own ~/.claude/state/ protected-path freeze are the same accident, and that neither source describes how it would be caught next time.
What we already know (from the vault)
- The failure mode is RDCO's own coinage, not borrowed. [[2026-10-24-coded-vs-prompted-agent-permission-gates]] phrases it "a correct rule with a wrong default breaking work while every log reports success," and names exactly one detection practice: check each affected target's mtime after each gate change, one gate at a time. That is a one-time post-change check, not a standing monitor.
- The observable signature is
success-on-a-denied-run. [[2026-10-23-rdco-auto-mode-classifier-miss-profile]] measured 251 denial events across 401 cron runs (2026-10-08 to 2026-10-23), withgraph-reingestandsync-contactsdenied in 15 of 15 runs each, while all 401 runs reportedsubtype: "success",is_error: false,safety_stops: 0.graph-reingest.logandsync-contacts-ledger.jsonwere frozen at mtime 2026-08-17. Detection lag was roughly two months. That brief declines to claim the blocks caused the full 67-day freeze, and this brief inherits that caution. - The mechanism is not fixable by a permission rule. Writes under
~/.claude/**are a built-in protected-path safety check that runs before settings allow-rules are evaluated ([[2026-10-24-coded-vs-prompted-agent-permission-gates]], verified against the Claude Code docs on 2026-10-24); the published exception list does not include~/.claude/state/. The documented workaround in [[2026-06-11-system-cron-runner]] is to write state via Bash heredoc, which is also what produced most of the 251 denials, because allow-listed commands only bypass adjudication when run standalone. - RDCO has already written down the wrong control once. [[2026-06-22-cron-auth-401-outage]] (a ~48h total cron blackout nobody caught) recommends "a watchdog that pings #ops/iMessage after N consecutive
is_error":trueruns." That is an exit-code watchdog. It would have caught the 401 outage and would not have caught the protected-path freeze, because the frozen runs all exited clean. - The nearest prior art already in the vault is a data-engineering note, not an agent note: [[2026-04-24-seattle-data-guy-silent-pipeline-failures]] describes a structurally identical mode (upstream stops sending, the job runs, succeeds on an empty delta, dashboards recycle last week's numbers until a human notices) and closes with "the scary failures aren't the ones that wake you at 2 AM, it's the ones that don't."
What the web says
- No source names this failure mode. Three searches (SRE silent-failure practice, permission dry-run/audit-mode practice, and an extended search on access-policy changes locking out service accounts) returned no incident writeup, pattern catalogue, or vendor doc that names "correct-rule-wrong-default" or any synonym for it as a class. The extended search returned explicitly: "The search turned up no specific incident matching your scenario." Treat the name as RDCO-internal.
- "Alert on absence" is real, named, and citable at the primitive level. Prometheus documents
absent()as "useful for alerting on when no time series exist for a given metric name and label combination," andabsent_over_time()as the same "for a certain amount of time" (prometheus.io, verified 2026-11-06). The practitioner name for the pattern built on it is dead man's switch / heartbeat monitoring: the job pings an external service on completion and the service alarms when the expected ping does not arrive. That family is extensively documented, but in my search almost entirely in vendor and DEV-community posts rather than primary SRE literature - flagging that source-quality gap rather than papering over it. - Heartbeat monitoring does not cover this case, and the practitioner literature says so. One of the cron-monitoring guides surfaced states the limit directly: a ping "catches the failure mode that pure heartbeat monitoring misses: the job that runs, succeeds by every technical measure, and still does nothing useful" - only if you also check the output. Single-sourced to a vendor blog, but it is the same conclusion RDCO's own
subtype: "success"evidence forces, so I am treating the convergence as the support rather than the blog. - Pre-enforcement staging is the mature, vendor-documented half, under at least two names. OPA Gatekeeper supports
deny,dryrunandwarn;denyis the default,dryrunmakes no changes and reports would-be-affected resources in the constraint'sstatus.violations, and--log-denieslogs deny/dryrun/warn failures together (open-policy-agent.github.io, verified 2026-11-06). Microsoft Entra calls its equivalent report-only mode: "a policy state that lets administrators test most Conditional Access policies before enabling them," evaluated at sign-in but not enforced, with results logged to the Report-only tab of the sign-in log; the doc frames it as "a valuable tool for validating policy changes progressively before you enforce them," and specifically advises creating a report-only copy of an enforced policy before modifying it (learn.microsoft.com, verified 2026-11-06). - Both tools ship the trap inside the fix. Gatekeeper's default enforcement action is
deny, so omittingspec.enforcementActionturns an intended observation rollout into live blocking. Entra's report-only mode does not evaluate the User Actions scope and never enforces grant or session controls, which the docs state plainly. A staging mode you can forget to set, or that silently covers less than you think, is itself a correct-rule-wrong-default surface. - The binding constraint on staging is window length versus job period, and that is the sharpest thing found. Entra's policy impact view offers 24-hour, 7-day and 1-month windows (verified in the doc above). A community guide surfaced in search made the matching point for the failure case: a policy rollout can miss "the service account that runs on the first of the month" because the job never fired during the observation window. That specific claim is a single community blog and is not verified against Microsoft documentation - flagged, not laundered. A widely-repeated "leave it in report-only at least a week" figure also appears in community guides and is not present in the Microsoft page I fetched; do not cite it as Microsoft guidance.
- Named candidates I could not verify this round (search and fetch caps were spent): SELinux permissive mode / permissive domains, AWS IAM Access Analyzer unused-access findings and custom policy checks, AWS IAM last-accessed (Access Advisor) data as a pre-tightening input, CloudTrail/CloudWatch metric-filter alarms on unauthorized API calls, GCP Policy Simulator, Kubernetes
ValidatingAdmissionPolicyaudit validation actions, and synthetic monitoring / synthetic transactions as a black-box probe of automated paths. Each is a real named thing I believe exists; none is cited as fact here, and the IAM group is filed as a follow-up.
Convergences and contradictions
- Convergence, and it is the finding. The web literature, the vault's data-pipeline note, and RDCO's own measured logs all land on the same split: liveness detection (did the job run?) and correctness detection (did the job produce its outputs?) are different controls, and this failure mode is invisible to the first. [[2026-04-24-seattle-data-guy-silent-pipeline-failures]] arrives there from stale dashboards, [[2026-10-23-rdco-auto-mode-classifier-miss-profile]] from a frozen ledger, and the cron-monitoring guides from "the job that succeeds and does nothing useful."
- Contradiction with a standing RDCO recommendation. The remediation banked in [[2026-06-22-cron-auth-401-outage]] - alarm after N consecutive
is_errorruns - is the correct fix for the 401 blackout and the wrong fix for the protected-path freeze. Both incidents are "a nightly path went dark and nobody noticed," so it is easy to assume one alarm covers both. It does not. This brief corrects that. - Contradiction with the literature's framing, in RDCO's favour. Every mature tool stages enforcement by evaluating the policy against live traffic and logging what it would have done. RDCO cannot do that for its own permission changes: the protected-path check is built into the harness and has no dry-run mode, and [[2026-10-24-coded-vs-prompted-agent-permission-gates]] established that
permissions.allowcannot pre-approve protected-path writes at all. So the dry-run family transfers as a discipline (stage, observe for a full period, then enforce) but not as a mechanism. Labeled as my inference. - Agreement that detection must be surfaced, not merely logged. [[2026-10-23-every-openai-dots-guide-always-on-agents]] names active progress-checking and a first-class visible surface as the fix for always-on-agent silent failures, and already flags that RDCO's channels agent relies on
ops-log.mdplus ad hoc iMessage instead. [[2026-08-31-monte-carlo-ai-observability-five-pillars]] is the counter-example to avoid: dimensions to look at, with no reference standard to check against.
Synthesis for RDCO
The answer is mixed and the shape of the mix is the useful part. There is no documented detection pattern for this failure mode as a class - no name, no catalogue entry, no incident writeup that treats "defensible default silently breaks unattended work" as a thing to design against. That negative is worth having, because it means RDCO should stop looking for the canonical answer and start composing one. What does exist, and exists in mature, vendor-documented form, is two separate named control families that happen to compose into the control this failure needs: alert-on-absence for after the fact (Prometheus absent_over_time, dead man's switch, heartbeat monitoring) and enforcement staging for before the fact (Gatekeeper dryrun/warn, Entra report-only mode). Nobody in the sources read connects them, and nobody addresses the case where the operator and the administrator are the same person and the protected path belongs to the harness rather than to the operator - which is the parent brief's first open follow-up, still open.
The expensive correction is that RDCO reached for the wrong half. The alarm already on record in [[2026-06-22-cron-auth-401-outage]] fires on is_error runs. Every run in the protected-path freeze exited clean: subtype: "success", is_error: false, safety_stops: 0, 401 times, while two state files sat frozen since 2026-08-17. A heartbeat is a liveness signal, and this failure mode does not kill liveness - it removes a side effect while leaving the process alive and the exit code zero. The control that matches the failure is freshness monitoring of the output artifact, not liveness monitoring of the job. The discriminating question for each nightly skill is not "did it run" but "is the thing it was supposed to touch newer than its period." Stated as my inference from the composition above, with the Prometheus absence primitive and the frozen-mtime evidence as the two supports.
Three concrete controls, cheapest first. (1) Reconcile the denial array the wrapper already receives. permission_denials arrives in the cron result JSON today and is discarded; have cron-job-runner.sh fail or at minimum annotate any run whose array is non-empty. This is already banked as a build task in [[2026-10-23-rdco-auto-mode-classifier-miss-profile]] and requires zero new data collection; I am re-ranking it to first because it is the only control that catches success-on-a-denied-run in the run that caused it rather than N periods later. It is also the missing instrumentation for [[feedback_automode_classifier_hard_gate]]: a hard gate whose denials are discarded has an unmeasurable compliance rate, which is exactly what the 10-23 brief found when ops-log.md was denied 37 times and written anyway. (2) A freshness manifest, checked daily, surfaced in morning-prep. One line per cron slug: job slug, the output artifact that proves it did its work, and a max staleness equal to roughly two periods. A bash check compares mtime against now and emits a single list of stale entries; morning-prep already reaches the founder daily, which converts weeks of lag into one day. Two design constraints, both inferences but load-bearing: the manifest and the monitor must live on a path RDCO owns rather than under ~/.claude/state/, or the monitor shares the failure domain of the thing it monitors; and reads under ~/.claude/ are not blocked, only writes, so the checker can inspect state files it cannot update. The artifact-per-slug mapping is the real work and some slugs may not have one - see follow-ups. (3) An enforcement-staging step for RDCO's own permission changes, since no dry-run mechanism exists. Before adding any deny or ask rule, or relocating any protected path: enumerate which cron slugs touch that path or verb, hold the change for one full period of the longest-period affected job, then check each affected target's mtime after. Step three is the parent brief's existing per-gate mtime discipline; steps one and two are the additions, and they come straight from Entra's documented practice of validating progressively against live traffic plus its 24h/7d/1mo impact windows. For RDCO that means a change touching deep-research can be judged the next morning, while one touching finance-pulse (first Sunday monthly) cannot be judged for a month - and shipping it anyway is a deliberate acceptance, not an oversight.
What this does not fix. None of the three controls catches a default that breaks a path with no durable artifact, and none of them catches the agent routing around a denial successfully, which [[2026-10-23-rdco-auto-mode-classifier-miss-profile]] showed happening on ops-log.md and cron-undelivered/. Control (1) is the only one that sees the routing-around, and only because the denial is recorded before the workaround succeeds. A monitor built on artifact freshness alone would have scored those two targets healthy.
Why this is in the vault
It closes open follow-up #2 of [[2026-10-24-coded-vs-prompted-agent-permission-gates]] with a mixed result that redirects RDCO's harness-safety build queue: it retires the exit-code cron watchdog recommended in [[2026-06-22-cron-auth-401-outage]] as insufficient for this failure class, re-ranks the permission_denials reconciliation from [[2026-10-23-rdco-auto-mode-classifier-miss-profile]] to the top of the queue, and specifies the freshness-manifest and enforcement-staging steps that were missing. It also records the negative result so no future /curiosity pass re-asks whether a canonical named pattern exists.
Open follow-ups
- Does any IAM vendor document a "check what was actually used before you tighten" workflow, and does it cover non-interactive and service identities? AWS IAM last-accessed (Access Advisor) data, IAM Access Analyzer unused-access findings and custom policy checks, and GCP Policy Simulator are all named candidates I could not verify this round. If one of them documents service-identity coverage, it is the closest thing in the industry to a pre-enforcement control for exactly this failure, and worth a night. Checked against
06-reference/research/: nothing there covers policy simulation or dry-run enforcement. - Which output artifact is the true work-proof for each RDCO cron slug, and what do you do with the slugs that have none?
graph-reingesthas a log,sync-contactshas a ledger, butcheck-boardandopen-threads-checkmay leave no durable trace on a no-op run, which makes "stale" and "nothing to do" indistinguishable. This is partly a build task; the research half is whether anyone documents freshness monitoring for jobs whose correct output is sometimes nothing. Unanswered in the research dir. - Is there published guidance on keeping the watchdog out of the watched failure domain? Prometheus Alertmanager ships a Watchdog alert and Dead Man's Snitch exists as a product; whether either documents the separation-of-failure-domain requirement, or RDCO is inferring it, is unverified. Worth a night only if control (2) gets built. Unanswered in the research dir.
- Does the Claude Code headless result JSON expose any success signal richer than
subtype/is_error- anything that distinguishes "completed having written its outputs" from "exited zero"?safety_stops: 0on 401 runs with 251 denials says the current signals are not trustworthy as a composite. If the harness exposes nothing better, that is itself the answer and it justifies the freshness manifest as permanent rather than interim. Unanswered in the research dir. - Is "the operator is the administrator" detection a distinct problem? Parent follow-up #1, still open, and this brief found no source addressing it. Every mature staging mechanism assumes an admin separate from the affected workload; a solo founder staging a change against his own agent has nobody to report the report-only findings to but himself.
Related
- [[2026-10-24-coded-vs-prompted-agent-permission-gates]]
- [[2026-10-23-rdco-auto-mode-classifier-miss-profile]]
- [[2026-06-22-cron-auth-401-outage]]
- [[2026-06-11-system-cron-runner]]
- [[2026-04-24-seattle-data-guy-silent-pipeline-failures]]
- [[2026-10-23-every-openai-dots-guide-always-on-agents]]
- [[2026-08-31-monte-carlo-ai-observability-five-pillars]]
- [[2026-07-02-queue-jsonl-task-distribution-primitive-spec]]
- [[2026-10-20-every-shared-agent-permissions-architecture]]
- [[feedback_automode_classifier_hard_gate]]
Sources
Web, primary, fetched and verified 2026-11-06:
- https://prometheus.io/docs/prometheus/latest/querying/functions/ -
absent()andabsent_over_time(), documented as "useful for alerting on when no time series exist" - https://open-policy-agent.github.io/gatekeeper/website/docs/violations/ - enforcement actions
deny/dryrun/warn,denyas the default,status.violationsreporting,--log-denies - https://learn.microsoft.com/en-us/entra/identity/conditional-access/concept-conditional-access-report-only - report-only mode definition, progressive-validation practice, report-only copy of an enforced policy, 24h/7d/1mo policy-impact windows, stated limitations (User Actions scope, no grant/session enforcement)
Web, search-summary level only, NOT fetched - cited as community-blog claims, not fact:
- The "job succeeds by every technical measure and still does nothing useful" limit on heartbeat monitoring (cron-monitoring vendor guides)
- The "service account that runs on the first of the month" miss and the "at least a week in report-only" figure (Conditional Access community guides; the week figure is absent from the Microsoft page above)
Named but unverified this round (no claim made): SELinux permissive mode, AWS IAM Access Analyzer, AWS IAM last-accessed data, CloudTrail unauthorized-API-call alarms, GCP Policy Simulator, Kubernetes ValidatingAdmissionPolicy audit actions, synthetic monitoring.
Vault:
~/rdco-vault/06-reference/research/2026-10-24-coded-vs-prompted-agent-permission-gates.md~/rdco-vault/06-reference/research/2026-10-23-rdco-auto-mode-classifier-miss-profile.md~/rdco-vault/08-tooling/2026-06-22-cron-auth-401-outage.md~/rdco-vault/02-sops/2026-06-11-system-cron-runner.md~/rdco-vault/06-reference/2026-04-24-seattle-data-guy-silent-pipeline-failures.md~/rdco-vault/06-reference/2026-10-23-every-openai-dots-guide-always-on-agents.md~/rdco-vault/06-reference/research/2026-08-31-monte-carlo-ai-observability-five-pillars.md~/rdco-vault/06-reference/research/2026-07-02-queue-jsonl-task-distribution-primitive-spec.md~/rdco-vault/06-reference/2026-10-20-every-shared-agent-permissions-architecture.md~/.claude/projects/-Users-ray/memory/feedback_automode_classifier_hard_gate.md