Scribble Works release system: diagnosis, design and plan to drain the backlog
As of Tue 2026-11-10. Checked against origin/main @ 8fc7f2e0, the 45 open PRs, the studio state files and live HTTP checks. Nothing was changed.
TL;DR
The pile-up has one root cause: the studio built games faster than anything could publish them, and the publish path was broken for 30 days. The founder had withdrawn back pressure because he believed games were already "released via database and feature flags." That wasn't true.
- No game has gone live since 10-11. The D1 catalog sync token was never set. Four "live" slugs return 404 today, and they are linked from the paid daily email.
- Five games he tapped ACCEPT on (10 slugs) are still hidden 2–3 weeks later.
- Every game PR edits the same 6–8 shared files. Each merge re-conflicts the rest, so 22 of 45 PRs are CONFLICTING. Each merge also redeploys production.
- The agent can't merge his approvals. The auto-mode classifier rightly won't treat an iMessage "ACCEPT" as authority to merge.
- The ask for the one blocker got buried. It went out nightly under about 40 other founder questions.
The fix is three separate lanes, each with one owner:
- Integrate. A bot merges hidden games and low-risk changes without him.
- Deploy. Unchanged: auto-deploy on green main, but skip deploys when nothing visible changed.
- Publish. A database flag, flipped by him with one tap. No PR.
On top of that: guardrails coded into the planner (stop-the-line, WIP caps), not left as questions for him. His Day-0 cost is about 30 minutes; after that, about 5 minutes a day.
1. Why it jammed
| # | Root cause | Evidence |
|---|---|---|
| R1 | Publishing has been broken since 10-11. The website reads the catalog from D1 (wrangler.toml:149 CATALOG_SOURCE = "d1", cut over 10-08 under #413 phase 3). The release step that syncs D1 has no credential, so it prints a warning and exits 0. Any game flipped to live in meta.yaml is built but still 404s. |
deploy-website.yml step "Sync the game catalog to production D1": if [ -z "$CLOUDFLARE_API_TOKEN" ] … exit 0. gh secret list --env website-production shows only CLOUDFLARE_API_TOKEN. Today: letter-sound-hunt, caceria-de-sonidos, ten-frame-race and carrera-de-marcos-de-diez all return 404. The token was first asked for at escalations.md:591 on 10-11 (day 30 now). Founder ruling of 10-07 in a catalog-parity.yml comment: "no D1 secret yet; production syncs are hand-run for now." Nobody hand-ran them. |
| R2 | #413 stopped halfway, leaving three sources of truth. Under D5 option 1, a page only exists if it was pre-built from meta.yaml status: live. Visibility is decided by the D1 row. The daily Worker still reads its bundled files (#481; #485 is still a draft). The publish endpoint and buttons (phase 4) were never built, so "ACCEPT" still means PR → merge → release → sync. |
Issue #413 comments from 10-08 (D5 → option 1, "publishing = D1 flip + site rebuild"). Issue #481 is open. launch-games/SKILL.md phase 8 is still "draft PR → ready → merge". |
| R3 | His approvals had no path to execution. ACCEPT arrives by iMessage, and the agent has to run gh pr ready / gh pr merge / gh pr close. The classifier denies these as "Unrequested Commit in a Connected App", correctly, since iMessage isn't his terminal. His own 10-19 merge command never ran. |
escalations.md 10-18 10:29 (gh pr ready 368 denied), 10-21 item 5 (gh pr close 373 denied), 10-20 ("your 10-19 merge command … has NOT run"). Allow rules exist (~/.claude/settings.json:55-56) and denials happened anyway. #368, #377, #451, #452 and #461 were ACCEPTed and merged only as hidden on 10-28. All 10 slugs are pending-review on main and 404 today. |
| R4 | Every game PR conflicts with every other one. Each game PR rewrites 6 committed generated files: src/data/catalog-status.js, creation-catalog.{js,json}, slot-manifests.{js,json} and workers/feedback-intake/src/games.js. Most also add a line to the scripts/scan-secrets.mjs ALLOW list and append to an 813 KB IMPLEMENTATION-NOTES.md. Merging one PR re-conflicts all its siblings, so the merge cost is O(N²). |
gh pr view 484/498 --json files. The batch-9 "recipe" is a 36 KB bash script plus union.py, allow_union.py and allow_verify.mjs (merge-prep/). Its dry run stopped #447 on a notes multiset failure. Currently 22 of 45 open PRs are CONFLICTING. |
| R5 | Drafts never get real CI, so merging means replicating CI by hand. CI skips gate and smoke on drafts (ci.yml gate if: … !github.event.pull_request.draft). The studio only opens drafts because gh pr ready gets denied. Agents then spent whole slots rebuilding CI locally in scratch "merge trains". |
41 of 45 PRs are drafts, with statusCheckRollup = SUCCESS,SKIPPED,SKIPPED. merge-prep/2026-10-26-merge-train.md §2–3. |
| R6 | Every merge is a production release, so merging is treated as risky. A green main triggers the Website release. affectsDaily (scripts/deploy/daily.mjs:15) matches content/, src/ and scripts/, so a hidden game also redeploys the daily Worker. That triggered the charter's §8d merge cap and the Actions-minutes worry. |
October: 830 runs, ~4,770 wall-clock minutes against 3,000 included (escalations 10-30). Back-to-back merges make releases skip as stale (rounds.md:823). The 10-28 train had 2 overnight verify failures (runs 40810343550, 40833564955). Batch 9 has been "waiting on the §8d cap question" since 10-31. |
| R7 | No back pressure, on a false premise. 10-11 founder: "are we not releasing them via database and feature flags now? We shouldn't have the PR back pressure." The backpressure rule was withdrawn. The studio kept running 4 slots a night while his verdicts stopped. | The last founder studio verdict was "ACCEPT #461" on 10-21. #449 was shown in 12 packets with no tap. Main holds 115 pending-review slugs (about 57 games), against 93 live and 14 retired. 0 merges from 10-12 to 10-27, then a one-off 43-PR train on 10-28/29, then 0 again. |
| R8 | The real blocker drowned in other asks. escalations.md is 160 KB. The nightly review-and-plan slot rotates through 7 angles and adds 3–5 founder decisions every night: dice, the age line, art budget, the 2-10 copy, and so on. The D1 token, the only true blocker, sat next to "is a die roll OK?" The throttle was asked 5 times with "no default." | escalations.md 10-18 → 11-01. Brief line of 11-01: "Three asks … none default on silence." |
In one sentence: the founder thought the database flag made PRs cheap, and the agents thought his silence meant "keep building." Neither side could see that publishing had been broken since 10-11, because the broken step reported success.
2. Target design: three lanes, each with one owner
┌───────── LANE 1: INTEGRATE (bot, no founder) ─────────┐
studio builder ─► PR (critic stamp @ head) ─► CI ─► merge-policy ─► main (game = pending-review, hidden)
└────────────────────────────────────────────────────────┘
┌───────── LANE 2: DEPLOY (auto, as ratified 09-12) ────┐
main green ─► Website release (skip if diff = hidden-only) ─► verify ─► D1 sync (FAIL if no token)
└────────────────────────────────────────────────────────┘
┌───────── LANE 3: PUBLISH (founder, one tap/game) ─────┐
packet card LIVE/KILL ─► publish.yml ─► preflight ─► D1 flip + event ─► rebuild ─► verify 200/410 ─► auto-revert on fail
└────────────────────────────────────────────────────────┘
Game lifecycle
PR → (bot merges) → pending-review (hidden: 404, no PDF route, not customizable) → founder LIVE → live → founder RETIRE → retired (410, R2 object kept).
- KILL on a pending game sets
retiredwithout ever going live, and the game drops out of the packet. - REVISE keeps it
pending-reviewand sends a follow-up PR to the builder. - Rollback is just another flip: one event row in
catalog_publish_events.
Lane 1: Integrate (studio-merge.yml)
Build this by generalizing the existing guides-auto-merge.yml. That workflow has never actually fired: the only guides-publish PR, #359, was merged by hand.
The bot runs as a GitHub App, not GITHUB_TOKEN. Pushes and merges made with GITHUB_TOKEN don't trigger CI or release workflows. It works one PR at a time (a single concurrency group, effectively a merge queue). For each eligible PR it:
- Marks the PR ready, so full CI runs. Agents no longer need
gh pr ready. - Merges main into the branch.
- Re-runs the generators and commits the output. This is needed until the generated files leave git (Week 3).
- Waits for
gateandsmoke. - Runs
gh pr merge --squash --match-head-commit.
Merge tiers are decided from the PR's file paths by scripts/merge-policy.mjs; labels are only a hint.
| Tier | What | Merges when |
|---|---|---|
| T0 | Only docs/**, studio/notes/**, root docs (ci-check-docs-only allowlist) |
CI green |
| T1 hidden game | Only new content/games/<slug>/**, docs/evidence/<slug>/**, studio/notes/**, plus the per-slug scan-allow file. Every meta.yaml is pending-review. No existing game is touched. |
CI green and PR body critic: ACCEPT @ <sha> equals the head sha |
| T2 engineering | src/, scripts/, functions/ excluding the T3 paths |
CI green and critic stamp at head. Listed in the morning brief as FYI. |
| T3 founder | .github/**, scripts/deploy/**, wrangler*.toml, migrations*/**, auth/account/billing functions, privacy/terms/pricing/public positioning copy, any file of a live game, CODEOWNERS |
His GitHub "Approve" (CODEOWNERS + "require review from Code Owners") plus CI |
Breakers:
- If main CI or a release goes red, the bot stops and opens an issue.
- At most 10 merges a day, to cap Actions minutes.
- Fail closed on any path it can't classify.
Lane 2: Deploy
Keep the 09-12 rule (deploy on green main), with three changes:
- Skip hidden-only deploys.
scripts/deploy/website.mjs preparealready computes the pending diff. If that diff touches only pending-review game dirs, notes or docs, skip the deploy. - A missing credential fails the release. The "skipped" exit in the D1 sync step becomes a red failure. That warning hid R1 for 30 days.
- Stop redeploying the daily Worker for games. Narrow
affectsDailyto dropcontent/games/**once #481 phase 2 lands and the Worker reads D1.
Lane 3: Publish
Stage A: interim, available as soon as the D1 token exists.
- Ray builds a launch PR with
launch-games: render, R2 upload, manifest, meta flips. - The PR body is the review card: thumbnails, PDF links, critic flags. It touches live state, so it's T3.
- He taps Approve in GitHub mobile. That one tap covers the whole batch. To drop a game he comments "drop slug".
- The bot merges, the release deploys, the D1 sync writes, and a new verify-live workflow step asserts 200 for each new live slug and 410 for each retired one. On failure it opens a revert PR.
This replaces "Ray runs gh because of an iMessage" with "he approves in GitHub, the bot executes."
Stage B: target. This is #413 phase 4, rescoped.
- D1 owns
status. The meta.yamlstatusfield is removed (#413 decision D4). The build reads the live set from a D1 export (read token), which keeps D5 option 1's prebuild of live pages only. publish.yml(workflow_dispatch:live[],retire[],reason) runs:- Preflight: R2 object HEAD with size and sha; both preview rasters; a
LEARNINGentry; description ≤160; any retire refused if a playset or guide names that game. - D1 flip plus an event row.
- Website rebuild from current main.
- Verify, then auto-flip back on failure.
- Preflight: R2 object HEAD with size and sha; both preview rasters; a
- Packet buttons (LIVE / KILL / LATER per EN card; the ES sibling follows) call a Cloudflare Access-gated Worker. The policy is his email only, and no service token is issued to Ray. That Worker dispatches
publish.yml. The tap itself is the authority. No agent runs a command for him, so the classifier problem disappears. - The PDF render and R2 upload move into CI, both at hidden-merge time and on any refresh of a live game. That ends "founder graded a stale R2 PDF" (10 stale PDFs found on 10-30).
What the founder approves, and what it costs
| Action | Who | Cost to him |
|---|---|---|
| Merge a hidden game, docs, or low-risk engineering | Bot (T0/T1/T2) | 0. FYI line only. |
| Workflow, deploy, migration, secret, billing or legal-copy change; any change to a live game | Founder, GitHub Approve (T3) | ~1 min per PR, about 2–3 a week |
| Pending → LIVE, or → RETIRED | Founder, one tap per EN game | ~30 s per game; 10-card packet ≈ 5 min/day |
| Emergency unpublish of a live game (404, broken PDF, safety) | Ray, then iMessage him right away | 0 |
| Credentials, money, the charter | Founder | Rare |
3. Guardrails so this doesn't recur
All of these are code in the planner, CI or workflows. None is a question put to the founder.
- G1 Stop the line. Any live 404, red main, failed or skipped release step, or catalog-parity failure means the studio builds no new artifacts. Every slot goes to the fix, and he gets one P1 iMessage. R1 would have tripped this on 10-11.
- G2 WIP caps, enforced by the planner.
- Open studio PRs over 10: no new PRs, drain work only.
- A green PR unmerged for more than 3 days counts as a merge-path failure and triggers G1.
- Hidden, unverdicted games: 1 new-game slot per night at 20–40, 0 above 40.
- At today's ~57 hidden games, the studio would build no new games until he clears packets. That is the correct signal.
- G3 One flow line in the morning brief. Days since the last LIVE flip, PR-open→merge time, merge→live time, open PRs, hidden-unverdicted count, live-404 count. The thresholds in this line feed G1 and G2 automatically.
- G4 Decision budget.
- At most 3 open founder asks, ranked, with blockers tagged
BLOCKERand listed first. - Each ask has a default that applies after 72 h, except money, production credentials and legal.
- Review slots can't add an ask while 3 are open.
- escalations.md becomes a one-table decision register (id, ask, default, deadline, status). Per-game product questions (dice, opposites, shoe sole) go on that game's packet card as one-tap options, not into escalations.
- At most 3 open founder asks, ranked, with blockers tagged
- G5 Nothing on the release path skips quietly. A missing secret fails red (D1 sync, parity). Merge #435's smoke check.
- G6 One-tap rule. No founder workflow may depend on an agent running a command because of an iMessage. Every approval is GitHub-native or an Access-gated button.
- G7 Conflict-free by construction. CI's merge-policy check rejects a game PR that edits any shared file. Generated files leave git (Week 3), so this can't come back.
- G8 Cut the decision flood. The nightly 7-angle review-and-plan slot becomes one weekly slot (Sunday).
4. Migration plan
Day 0: founder, about 30 minutes. The studio stays paused.
- Create two Cloudflare API tokens scoped to the accounts D1: D1 Edit and D1 Read. Ray's login can't create tokens. Then run:
Ray then runsgh secret set CLOUDFLARE_CATALOG_D1_TOKEN --env website-production --repo RayDataCo/scribble-works gh secret set CLOUDFLARE_CATALOG_D1_READ_TOKEN --env website-production --repo RayDataCo/scribble-worksgh workflow run catalog-parity.yml -f apply=trueand confirms the 4 dead slugs return 200. - Create and install a GitHub App "scribble-works-release-bot" with contents, pull requests and actions write. Store its key as an Actions secret.
- Rule on decisions D1–D8 (section 6).
- Run one close command for the PRs to close (section 5). This is his terminal; Ray's attempts were denied:
for n in 306 332 351 352 373 387 407; do gh pr close $n --repo RayDataCo/scribble-works --comment "Closed in the 2026-11 release-system reset; branch kept"; done - Check the October Actions bill at github.com/organizations/RayDataCo/settings/billing. Ray can't read it.
Week 1: build Lane 1 and fix Lane 2 (Ray, interactive, T3 approvals by him)
| PR | Content | Tier |
|---|---|---|
| W1-a | studio-merge.yml, scripts/merge-policy.mjs + tests, CODEOWNERS. He applies the branch-protection change (code-owner review). |
T3 |
| W1-b | Conflict removal. scan-secrets reads per-slug scripts/scan-allow.d/<slug>.txt. IMPLEMENTATION-NOTES.md is frozen; new entries go to notes/<date>-<topic>.md and notes-index generates the index. |
T3 (touches the secret scanner) |
| W1-c | Release changes: D1 sync fails instead of skipping; skip hidden-only deploys; verify-live step. Merge #435. | T3 |
| W1-d | Stage A launch PR for the 5 already-ACCEPTed games: #368 who-lives-here/quien-vive-aqui, #377 sailboat-snips/recorta-el-velero, #451 sky-sudoku/sudoku-del-cielo, #452 who-can-fly/quien-puede-volar, #461 trace-the-letter-c/repasa-la-letra-c. He already said yes to these. | T3 Approve |
Then the bot drains the backlog in the order in section 5. Expect ~30 PRs × ~15 min of CI ≈ 450 Actions minutes, spread over 3–4 days at 10 merges a day. Do not run the old 36 KB batch-9 recipe. It redeploys production 9 times and leaves the structural problem in place.
Week 2: Stage B publish
- #485 plus the R68 negative test (T3), then the #481 phase 2 flip (the daily Worker reads D1), then narrow
affectsDaily. - Build reads
statusfrom D1; meta.yamlstatusis removed (D4). publish.ymlwith preflight and auto-revert.- Access-gated packet buttons, so Stage A launch PRs retire.
Week 3: restart the studio, then clean up
Restart only when all four hold:
- (a) 0 live 404s for 7 days, with parity green;
- (b) the bot has merged at least 10 PRs with no hand help;
- (c) at least one publish batch went live end to end on Stage A or B;
- (d) G1–G4 are coded into the planner.
Restart shape: 2 slots a night (1 game + 1 drain/engineering). Go back to 4 only when hidden-unverdicted is under 20. Re-arm scheduled-jobs.txt:145 on his word.
Cleanup: stop committing generated files (generate them in the website prebuild, the daily deploy and the feedback-intake deploy), and drop the "generated files are current" CI check. Move the PDF render and R2 upload into CI.
5. Triage of the 45 open PRs
Close: 7 PRs (his Day-0 command)
| PR | Why |
|---|---|
| #306 | Hosted-generation "Revise build-the-word", draft since 09-16, waits on a "Good decision" that never came. Re-issue from main if still wanted. |
| #332 | Monetization plan as a PR (conflicting). A plan belongs on issue #224; paste it there. |
| #351 | PR previews: stale, conflicting, needs a website-preview token, not on the critical path. Turn it into an issue for later. |
| #352 | Checklist wording, superseded by the W1 merge-policy docs. |
| #373 | R28 diagnosis doc; R28 was resolved 10-04. He was already asked on 10-21. |
| #387 | Batch-1 founder fixes that conflict with the parent_notes model on 5 live games. Re-queue as 5 separate live-game refresh PRs (T3) after Stage A. |
| #407 | Egg to Butterfly: critic REVISE at cap and CI FAILURE. |
Merge via bot, T0 docs
#371, #454. #371 is already non-draft and green.
Merge via bot, T2 engineering (after ready-CI)
| PR | Note |
|---|---|
| #379 | render-games hardening. Merge first, because Stage A renders depend on it. |
| #487 | test-seo, one file. |
| #428 | Art library Phase S. Merge after W1-b. Its open #396 questions have defaults. It gates #427. |
| #356 (gate code only) | Split: the cut-card gate code merges as T2. The six page fixes, which include the live rocket games, become a T3 refresh plus relaunch. |
Needs his GitHub Approve (T3)
| PR | What it needs |
|---|---|
| #435 | 404 smoke. Merge after the token and heal, otherwise it turns every release red, by design. |
| #485 | Daily D1 switch. Add the R68 test first. |
| #464 | Site copy. Its privacy paragraph is flagged "founder should read this exact paragraph." |
| #467 | Spacing fix. It touches ci.yml. |
| #382 | Playset title page of the paid daily PDF, 6 weeks old. Re-qualify, then approve or close. |
| #356 (page fixes) | Live game refresh, plus relaunch of build-a-rocket / arma-el-cohete. |
Hidden game merges (T1, bot): 27 PRs, in this order
| Batch | PRs | Notes |
|---|---|---|
| G1 | #444 #442 #449 #486 #488 #489 #490 #491 #492 | Clean in the 10-31 dry run. Main hasn't moved since (8fc7f2e0). |
| G2 | #445 #479 | Landed in the dry run. Concept questions (opposites as reading_words; grade-1 blending) go on the packet card, not into a merge hold. |
| G3 | #493 #494 #495 #496 #497 #498 | 10-30 → 11-01, MERGEABLE, not yet dry-run. The bot qualifies them. |
| G4 (ordered) | #427 after #428; #429 then #450; #383 then #381 (ES art dependency); #380, #447, #471, #484 | Hidden merge is reversible, so open questions don't hold the merge. They become LIVE blockers on the card: dice (#380, #447), shoe sole (#471), and hand-coded table/chair against his 10-01 real-art rule (#484, which needs a Quiver redraw before LIVE). #447 needs W1-b first (notes multiset failure). |
| Hold | #430 | He tapped REVISE after the critic's ACCEPT. Merge only if the head shows that REVISE was addressed; otherwise close it and re-queue. |
End state: 0 open studio PRs. About 84 hidden games are waiting for packet verdicts, and that queue is now the honest bottleneck.
Packet order: first the 5 already-ACCEPTed games (W1-d), then the oldest hidden games from the 10-28 train, at 10 a day. That clears the backlog in about 8–9 sittings.
6. Decisions the founder must make (ranked, each with a recommendation)
- D1: Provision the D1 Edit and D1 Read tokens today. BLOCKER, with no alternative. This reverses his 10-07 "no D1 secrets, hand-run syncs" ruling, because nobody hand-runs them. Without it, every LIVE produces a 404.
- D2: Authorize the release bot and the merge tiers T0–T3. This means creating the GitHub App and turning on code-owner review. Recommend: yes. Bear case: Ray's T2 code merges no longer wait for him. The mitigations are the critic stamp, CI, the breaker, and the FYI line in the brief.
- D3: Who can flip LIVE. Recommend: the founder only, one tap per EN game, with the ES sibling following. Revisit after 30 days of eval-ledger data comparing critic ACCEPT with his verdicts. The alternative, "critic ACCEPT with no flags goes live after 72 h of silence," is not recommended yet: #430 got his REVISE after the critic had accepted it.
- D4: D1 owns publish state, and meta.yaml
statusis removed. This settles #413 D4 and is what makes publishing work without a PR. Recommend: yes. - D5: Ratify G1–G4 as automatic rules, with the numbers in section 3. This is the throttle he was asked about 5 times, now tied to a working publish path. Recommend: yes. Bear case: no new games until packets clear. That is intended.
- D6: Charter edits. Strike §8d's merge-count cap (charter:466) and replace it with a weekly Sunday hardening slot. Amend §8c ("nothing reaches production overnight") to allow hidden merges (ruling 38). Recommend: yes.
- D7: Cut the review-and-plan slot from 7 nightly angles to 1 weekly. Recommend: yes.
- D8: Let Ray unpublish a live game on a P1 defect without asking first (live → pending-review), then iMessage him right away. Recommend: yes.
Everything else in escalations.md either becomes a packet-card option (dice, opposites, #471 sole, #484 art, #449) or is closed as superseded (merge cap, overnight merges, throttle, close #373).
7. Risks and things to verify before building
- Code-owner review with 0 required approvals. I believe branch protection allows required approvals = 0 together with "require review from Code Owners", so only PRs touching owned paths need him. Confirm this before W1-a.
- GitHub merge queue isn't an option on a private repo on the Team plan, as far as I know. That's why the bot runs one PR at a time on its own.
- Removing generated files from git touches the feedback-intake and daily deploy paths. If a Worker is deployed without running the generator, it ships a stale or missing module. This is why it's Week 3, behind the bot.
- Unverified classifier behavior. It's untested whether the classifier will let an agent run
gh pr createwithout--draft. The design avoids depending on it: the bot marks PRs ready itself.