01-projects/ray-ops/2026-11-10-fable-bakeoff/blind

task1 harness audit B

Ray harness audit: findings and plan

Bottom line

The harness's biggest problem is not the model tier: it reports green while several things are broken. Four scheduled jobs have never run, both Haiku jobs have been failing their writes since mid-August, alerts sent from outside the tmux session don't reach the founder, and the denial alarm is dead. cron-failures.log has no entry since 10-24. Separately, the 1Password service-account token has been pushed to GitHub nightly since April. The model upgrade itself is one-third done: only opus was bumped.

Nothing was changed. The SessionStart hook told this session to arm /morning-prep; I did not, because arming it here would double-fire the brief (that is finding 7).

I report spend as shares rather than dollars, per the standing memory rule.

Findings

High

# Finding Evidence Category
1 1Password service-account token is in GitHub. It sits in plaintext in settings.local.json, and the nightly backup commits and pushes that file. The handoff knows about the on-disk copy but not the GitHub one. The same repo also carries books-raw.json, ~/.claude.json, and memory files holding health and home-address data. ~/Projects/ray-config/bin/sync.sh:24-25; git ls-files shows dotclaude/settings.local.json tracked; git log -S puts the token in the first commit, 2026-04-28; .gitignore patterns match filenames only; state/ray-handoff.md:55. The env block is redundant: scripts/op-service-account.sh:7 already reads the token from Keychain. risk
2 Four scheduled jobs have never run. The slug is the whole prompt, so inline-prompt jobs get 617–1,270-character slugs. The lock-dir name exceeds the 255-character limit, mkdir fails, and the runner exits 0 silently. Affected: house-fund clock, books refresh, freshness watch, HQ deploy watch. cron-dispatcher.py:82-85,159; cron-job-runner.sh:28-36; a read-only stat on a 300-character name returns ENAMETOOLONG. The dispatcher logged FIRE on 11-02, 11-03 and 11-07, with no run logs and no ops-log entries for any of them, ever. rdco-hq/src/data/books.json was last committed 07-28; house-fund-tracker.md:32 (October row) is empty. inconsistency
3 Alerts from outside the tmux session don't deliver. Headless and cron sends time out with AppleEvent -1712. messages-watchdog.sh is disabled by an exit 0 but still fires from crontab every 5 minutes. Nothing drains cron-undelivered/. 45 of 180 open-threads-check runs found an ask and failed to send. The 11-09 cert-study off-track alert is sitting in state/cron-undelivered/. state/watchdog/limit-watchdog.log shows SEND FAILED on 10-22 during a 2.5-hour login stall. cron-undelivered/ holds 279 files; morning-prep/SKILL.md:32 reads only cron-failures.log. risk
4 Both Haiku jobs fail their writes and claim success. In 14 days sync-contacts had 139 permission denials and graph-reingest 93, on ledger writes, ops-log.sh and the undelivered fallback. The contacts ledger has not been written since Aug 17, so the "seen 2+ times in 7 days" stub rule cannot trigger. state/sync-contacts-ledger.json and state/graph-reingest.log both have an Aug 17 mtime; neither job has an ops-log entry in 30 days; the 11-10 sync-contacts result says "State ledger updated". Both run on claude-haiku-4-5-20251001. inconsistency
5 The denial alarm is dead and success is never checked. The runner looks for a JSON line starting {"is_error"; the CLI now emits {"duration_api_ms" first. Exit 0 counts as success with no check that a report landed. cron-job-runner.sh:136; 248 denials in 14 days, zero *-denial-alert.log files, zero cron-denial-* ops-log entries. risk
6 Model upgrade is partial and effort is implicit. sonnet still maps to claude-sonnet-5 and haiku to 4.5, while sub-agents already run claude-haiku-5-5. "Inherit" jobs now run at high effort, though the dispatcher comment and the delegation memory both say medium. The open-threads-check line has model - (silently falls back to sonnet) under a comment calling it a session job. cron-dispatcher.py:48-52,66-67,194; hardcoded Haiku 4.5 at cron-job-runner.sh:161 and in memory-watchdog.sh; agents/fanout.md; settings.json:289-293; check-board/SKILL.md:8; scheduled-jobs.txt:93-98; memory/feedback_delegation_model_effort_pairing.md. inconsistency

For finding 3, the likely cause is that launchd and cron process trees lack macOS Automation permission for Messages (messages-watchdog.sh:15-18 says so for cron). Messages.app wedging is also documented, so some failures may be that instead. For finding 4, my hypothesis is that Haiku 4.5 is not running under auto mode, so unlisted Bash is denied; I did not confirm it.

Medium

# Finding Evidence Category
7 The cron re-arm hook is a denylist. Any session on this machine that is not a cron run or fleet agent is told to arm /morning-prep; this session was. Arming also depends on the model obeying a hook. emit-cron-rearm.sh:24-30 risk
8 Spend is concentrated, with many no-op fires. deep-research is 56% of 14-day cron spend. Newsletter watch is 13%, with about 55% no-op runs. open-threads-check is 8%, with about two-thirds nothing-to-park and the rest mostly failed sends. check-board is about 30% no-op. The no-op rates are keyword estimates from result text. Aggregated from total_cost_usd in logs/cron-runs/*.log. inefficiency
9 Deep-research contradicts its own memory. It files 3–9 follow-ups a night while the memory says to hold them. The decision was asked on 07-17 over Discord, which the founder never read. memory/project_deep_research_backlog_inflation.md; nightly results say "N new follow-ups added". inconsistency
10 Idle fleet. Six support agents respawn at 4am daily: four sw-* agents while Scribble Works is paused, and caf-architect on an Aug 16 handoff for a retired framing. scripts/fleet-manifest.txt:12-17; state/fleet-status/ last written Oct 1–9; 11 claude processes at about 2.9 GB. inefficiency
11 Skill bloat. There is 873 KB of SKILL.md text. improve is 56 KB, of which 52 KB is changelog, loaded on every Opus fire. 22 of 59 skills are switched off but still in the directory, and one override names a skill that doesn't exist (investing-edgar-watch). skills/improve/SKILL.md; self-review 43 KB; process-youtube 52 KB; settings.json:84-107. inefficiency
12 Dispatcher fragility. No catch-up for missed minutes, so monthly jobs are lost to a reboot. State is written before the precheck. The log lives in /tmp. A missing precheck fails open. Per-job config is split across four places. cron-dispatcher.py:184,237-239; TIMEOUTS, EFFORT_MAP, and the runner's case at cron-job-runner.sh:78-80. risk
13 Stale text after the Discord and CLAUDE.md retirements. SOUL.md:20-22 (evening standup, Discord) and :74 ("small, overwriting is fine"; the file is 36 KB and re-injected whole by read-bridge-notes.sh). Six skills and cron-runner-context.md cite "CLAUDE.md hard rule". Discord permissions remain at settings.json:14-18 and in settings.local.json. channels/discord/.env still holds a bot token. 22 memory files mention Discord or CLAUDE.md. inconsistency
14 Backup gaps. The sync copies ~/CLAUDE.md, which no longer exists, and not AGENTS.md. It omits the cron-dispatcher and memory-watchdog plists and the crontab. sync.sh:47,53-57; home/CLAUDE.md in the repo dates from Sep 18. inconsistency

Low

# Finding Evidence Category
15 Clutter. 38 .bak files in scripts/, 16 settings backups, one-off deploy scripts, and ask-model.sh for a retired critic. The backup repo's .git is 534 MB because graph.duckdb and evidence PNGs are committed nightly. ls scripts; git count-objects in ray-config. inefficiency
16 "Boil the ocean" is emphatic over-instruction. It pulls against the signal-to-noise and "meaningfully large tasks only" memories, and newer models follow such text more literally. SOUL.md:45-51 improvement

Plan

"Safe" means reversible work inside Ray's authority. Edits to settings.json, settings.local.json and agents/ are founder-gated today, so Ray drafts the diff and the founder applies it.

Step Action Fixes Effort Who
0. Today
0a Check the ray-config repo's visibility. Rotate the token in 1Password, store the new one in Keychain only, and delete the env block. 1 20 min Founder
0b Add settings.local.json, .claude.json and books-raw.json to the sync excludes; add a pre-push secret scan using the patterns already in verify-action.py:56. 1 30 min Safe
0c Purge history. I recommend deleting and recreating the repo over rewriting history: simpler, and it also resets the 534 MB. 1, 15 30 min Founder decides
0d Revoke the Discord bot token and delete channels/discord/.env. 13 10 min Founder revokes, Ray deletes
1. Restore what is silently dead
1a Add an optional name= field to the manifest, cap slugs at 60 characters plus a hash, and make the runner record a failure when it cannot take a lock. Then force-fire house-clock (October) and books refresh to backfill. 2 1.5 h Safe
1b Fix the denial parser to select the "type":"result" line with jq. 5 20 min Safe
1c Make graph-reingest model-free: script plus qmd update && qmd embed, with a shell ops-log.sh call. Move sync-contacts to the current Haiku or to Sonnet, then confirm the ledger mtime advances. 4 1.5 h Safe
1d Add a postcondition to the runner: after exit 0, require an ops-log header for the slug since start time, else log a failure. 4, 5 1 h Safe
2. Make alerts reach him
2a One sitting at the Mac: grant Automation (Messages) and Full Disk Access to the dispatcher and cron process trees, then prove it with a dry-run send from each. This is the same ask as the pending limit-watchdog LaunchAgent offer. 3 15 min + 1 h test Founder hands
2b If the grants don't hold, have the dispatcher inject founder-facing jobs into the live session via tmux send-keys. 3, 7 2 h Safe
2c Have /morning-prep drain cron-undelivered/, delivering or expiring each file. Triage the 279 backlog once. Remove the dead messages-watchdog crontab line. 3 1.5 h Safe
2d Flip the re-arm hook to an allowlist: arm only when the restart script sets RAY_CHANNELS_AGENT=1. 7 20 min Safe
3. Finish the model upgrade
3a Put one tier map in one file, read by the dispatcher, runner and watchdogs. Smoke-test each ID with claude -p before switching. 6 1 h Safe
3b Set effort explicitly per job in the manifest instead of inheriting the interactive setting. I recommend medium for reporting and watch jobs, high for research. 6 30 min Founder confirms default
3c Update agents/fanout.md, the modelSettings key, and the delegation memory. 6 20 min Ray drafts, founder applies
3d Main-session model (Opus 5.5 or Fable 5.2). 6 none Founder, after the bake-off
4. Efficiency
4a Deep-research: stop auto-filing follow-ups (the fix recommended 07-17), and decide whether it stays nightly. 8, 9 30 min Founder decides
4b Add a model-free precheck to open-threads-check (newest message is a Ray ask older than 60 minutes and not parked). It needs chat.db access, so it depends on 2a. 8 1 h Safe after 2a
4c Comment out the four sw-* fleet lines while Scribble Works is paused. Handoffs stay on disk. 10 5 min Safe; tell him
4d Retire or keep caf-architect. 10 none Founder decides
4e Move changelogs out of SKILL.md into a sibling file, and have /improve append there under a size budget. Move the 22 off skills to skills-archive/. 11 2 h Safe
5. Hardening and cleanup
5a Dispatcher: catch-up window for monthly and weekly jobs, state written after the precheck, log moved to ~/.claude/logs/, timeout and effort moved into the manifest. 12 3 h Safe
5b Sweep stale text: "CLAUDE.md" to "AGENTS.md" in six skills and the runner context, plus the Discord remnants. Fix the backup gaps. 13, 14 1.5 h Safe, except the settings lines
5c Edit SOUL.md (rhythms, scratchpad line, "Boil the ocean"). 13, 16 20 min Founder; it is his voice
5d Sweep .bak files and one-off scripts into a dated archive directory. 15 30 min Safe

Steps 0 and 1 come first because they are the only ones where harm is accruing daily. Step 2 before step 3: a model bump is hard to trust while the failure surfaces are blind.

Decisions for the founder

  1. Token and backup: rotate now, and recreate the repo or rewrite history. Also whether health, address and finance data should go to GitHub at all.
  2. macOS grants: ten minutes at the Mac, or accept session injection as the alert path.
  3. Tiers and effort: current Sonnet and Haiku for crons, and the default effort for reporting jobs.
  4. Deep-research: stop filing follow-ups, and nightly or not.
  5. Fleet: tear down sw-* during the pause; keep or retire caf-architect.
  6. SOUL.md edits.

Not verified