autoreview teardown — a shipped second-model critic, and the four things it does better than our plan
Why this is in the vault
The founder sent this link without comment on 2026-07-26, one day before the parked Codex-critic work comes off the parking lot. It is a working, public implementation of the exact thing we planned to build in [[2026-07-25-multi-agent-claude-codex-grok-composition-patterns]]. That makes it the cheapest available correction to our own design: instead of discovering the failure modes by shipping, we can read what someone already hit.
Verdict given: read, don't install, steal the design. The four amendments below now belong to the Monday plan.
What it is
openclaw/agent-skills → skills/autoreview. A pre-commit/pre-ship code review skill that shells out to a second model (Codex gpt-5.6-sol default at high reasoning, Claude claude-fable-5 optional, Pi third) and returns structured findings.
Provenance, checked 2026-07-26: MIT, organization-owned, 1,006 stars, 78 forks, created 2026-05-22, pushed same day as reading. Young but actively maintained. Not first-party OpenAI or Anthropic.
Security review (standing policy: review before install, even unasked)
The engine is scripts/autoreview, a single 12,626-line Python file. Full line-by-line audit was not performed and this note does not claim one. High-risk surfaces checked:
| Surface | Finding |
|---|---|
| Outbound network | None in-process. urllib imported for .parse only — no urlopen, no requests, no socket. The single http literal is a JSON $schema reference. Egress happens only via the engine CLIs it invokes, which is inherent to the design. |
| Subprocess / exec | 47 call sites, all consistent with shelling out to review engines. |
| Environment handling | Explicit allowlist (AUTOREVIEW_PROVIDER_ENV_ALLOW) rather than blanket inheritance; strips process-injection, Git-override, and credentialed-proxy values from the reviewer subprocess. This is sandbox construction, i.e. a positive signal. |
| Auto-install behavior | Refuses to auto-install its TruffleHog dependency; fails with a link instead. |
Nothing alarming surfaced. Recommendation is still not to install it: it is wired for a Codex/Guardian/clawsweeper ecosystem RDCO does not run, and the first-party openai/codex-plugin-cc remains the smaller install surface for our purposes. Take the design, not the code.
The four amendments to our plan
1. Scan the diff for secrets before it leaves for the third-party model — and fail closed
"Before engine invocation, autoreview runs TruffleHog over temporary snapshots of the exact added, modified, or deleted content under review… if one of those deleted values also occurs in added, context, or mixed staged/unstaged content, the review fails closed." (SKILL.md L45)
This is the real find, and it is a gap in our 2026-07-25 plan. Wiring Codex as a critic means shipping RDCO and phData-adjacent diffs to OpenAI. We already hold a hard [[feedback-no-secrets-on-disk]] posture and run a leak gate on the public plugin surface; we had no equivalent gate on the outbound-to-model path. Note the refinement worth copying: it matches TruffleHog's low-false-positive pre-commit policy (verified,unknown) rather than flagging every password-shaped string, so the gate stays usable.
2. Default the output to P0 only
"Default output is P0 only: report issues worth blocking the current change" (L24)
This is a direct structural answer to the failure mode our own research flagged as the top risk — multiple independent reports of ~8 findings per review regardless of code quality, and one case of Codex inventing plausible concurrency bugs that cost 30 minutes to disprove. We had identified the disease and prescribed vigilance. They changed the default. Changing the default is the better fix.
3. Convergence loop, but scope-bounded
"Keep going until structured review returns no accepted/actionable findings only while the work remains inside the original task scope." (L35)
Our research surfaced a convergence rule from one practitioner ("not done until Codex returns clean 3 consecutive runs") but no scope guard. Without one, a critic loop ratchets: each pass invites a slightly wider fix, and the change balloons. The scope clause is what makes the loop terminate on something recognizable as the original task.
4. Never silently switch engine or model
"Never switch or override the requested review engine/model except for the documented Codex Sol-to-Terra account-access fallback. Capacity, rate-limit, and unrelated failures keep the same engine/model." (L38)
A silent downgrade produces a review whose provenance is wrong, which is worse than no review. Maps onto our existing [[feedback-workflow-agent-output-integrity]] scar tissue about agents reporting work they did not do.
The idea we have no version of at all
They pair autoreview with a source-blind behavior-validator:
"Autoreview is source-aware and judges the change bundle; behavior validation is source-blind and judges the running product or tool against a behavior contract. A clean autoreview is not proof that a UI, CLI, API, or generated artifact works from the user's perspective." (L11)
Every critic in the RDCO fleet reads the source — /design-critic, /verify-vault-write, /verify-strategic-output, /video-critic, /verify-pdf-output, /supervise. All of them judge the artifact or its inputs. None judges the running thing against a stated behavior contract, blind to how it was built. That is a genuinely missing axis, and it is the axis that catches "the code is correct and the feature still doesn't work."
Worth noting the closest thing we have is /video-critic, which scores a rendered video against its locked storyboard — that is contract-vs-output, and it is the model to generalize from.
Corroboration worth recording
Two of their principles were reached independently by our 2026-07-25 research before we saw this skill:
- Advisory, never authoritative. Theirs: "Treat review output as advisory. Never blindly apply it. Verify every finding by reading the real code path" (L28-29). Ours: "a cross-model CRITICAL is a HYPOTHESIS the parent verifies against primary source, never an action trigger."
- No nested reviewer panels. Theirs: "Do not invoke built-in
codex review, nested reviewers, or reviewer panels from inside the review" (L48). Ours: one gate per chain must hit the primary source.
Independent convergence on the same two rules raises confidence in both. It does not make either verified — both are still design judgment, not measured outcomes.
Related
- [[2026-07-25-multi-agent-claude-codex-grok-composition-patterns]] — the parent research; this note amends it
- [[feedback-workflow-agent-output-integrity]] — the seven observed factory failure modes
- [[feedback-mcp-install-security-review-default]] — why the security section above exists
- [[feedback-fresh-eyes-subagent-for-own-artifacts]] — why RDCO runs critics at all
Sources
openclaw/agent-skillsskills/autoreview/SKILL.md, read 2026-07-26 (35,513 bytes). Line numbers cited above are from that read.scripts/autoreview, 12,626 lines, inspected at the risk surfaces named in the security table — not audited in full.- GitHub repository metadata via API, 2026-07-26.