A few random notes from claude coding quite a bit — @karpathy

Why this is in the vault

The multica-ai/andrej-karpathy-skills repo (Forrest Chang) is trending hard — a 4-rule CLAUDE.md distilled from this post, 100k+ GitHub stars. The repo is the compression; this note is the source. Worth filing because RDCO runs a large code-writing skill surface (audit/pipeline/execution scripts) and Karpathy's actual failure-mode taxonomy is a useful lens to audit our own agentic-coding discipline against. We file the source, not the wrapper, so we map against Karpathy's real argument rather than someone else's four bullets.

The core argument

The post (2026-01-26, single long-form X note titled "A few random notes from claude coding quite a bit last few weeks") is a sprawling ~10-section field report, NOT a tidy failure-mode list. Karpathy reports going from ~80% manual+autocomplete / 20% agents in November to ~80% agent / 20% touchups by December — "mostly programming in English now," which he calls the biggest change to his workflow in ~2 decades.

The failure observations all sit inside one paragraph ("IDEs/agent swarms/fallibility"). His frame: the mistakes have changed — no longer syntax errors, now "subtle conceptual errors that a slightly sloppy, hasty junior dev might do." Specifically:

• Wrong silent assumptions. The most common category: models make wrong assumptions on your behalf and run with them without checking. They don't manage their confusion, don't seek clarifications, don't surface inconsistencies, don't present tradeoffs, don't push back when they should, and are still a little too sycophantic. (Plan mode helps; he wants a lightweight inline plan mode.)
• Overcomplication / abstraction bloat. They overcomplicate code and APIs, bloat abstractions, don't clean up dead code. He gives the concrete example of a 1000-line brittle construction that collapses to ~100 lines the moment you ask "couldn't you just do this instead?" — and the model cheerfully agrees.
• Collateral edits. They still change or remove comments and code they don't like or don't understand, even when orthogonal to the task.
• Notably, he says all of this persists "despite a few simple attempts to fix it via instructions in CLAUDE.md" — i.e. he is skeptical that a prompt file fully solves it. (Which is mildly ironic given the repo built on top of him.)

His positive recommendation lives in a separate "Leverage" section, and it is the load-bearing prescription: LLMs are exceptionally good at looping until they meet specific goals. His phrasing — give it success criteria and watch it go, rather than telling it what to do step by step. Concrete tactics: get it to write tests first then pass them; put it in a loop with a browser MCP; write the naive-obviously-correct algorithm first then ask it to optimize while preserving correctness; shift from imperative to declarative to get longer agent loops and more leverage. Other sections (tenacity, speedup-vs-expansion, fun, skill atrophy, "slopacolypse") are observations, not coding rules.

Chang's wrapper (what the repo did to it)

multica-ai/andrej-karpathy-skills (also mirrored under forrestchang) is a single MIT-licensed CLAUDE.md that compresses the above into four named, prescriptive rules: Think Before Coding, Simplicity First, Surgical Changes, Goal-Driven Execution. Useful as a starter CLAUDE.md for a bare repo, but for us it is low marginal value — RDCO's CLAUDE.md and skill harness already encode stricter versions of all four, and the repo adds no tooling, only prose. Do-not-install (it's a file to copy, not a dependency; nothing to run, security-clean by construction — no scripts, no hooks, no network). Honest finding worth recording: the four-rule framing is Chang's construction, not Karpathy's. Karpathy named no rules and no four-part taxonomy; he wrote a discursive field report and Chang extracted/relabeled three failure observations plus one "Leverage" tactic into four imperative headings. The "source" as commonly cited is partly synthetic.

Mapping against Ray Data Co

Where the discipline actually bites for us — honest, including where we likely fall short:

1. Goal-driven / success-criteria — we are strongest here, and it's deliberate. audit-newsletter-outputs.py is a pure-Python, zero-LLM-call invariant suite (I1–I13) gating /process-newsletter outputs. That IS Karpathy's "give it success criteria and watch it go," implemented as a deterministic check that cannot hallucinate a pass. The pipeline seats (spec → tests → code → critic) bake tests-before-code in by construction. This is the one rule we can claim we satisfy structurally rather than aspirationally.

2. Surgical changes — we likely violate this most in practice, and our own workflow encourages it. The feedback_proactive_vault_contributions norm ("create concept articles and cross-links without asking permission") plus auto-mode's bias-to-action is precisely the "improve neighboring code / touch files you weren't asked to" failure Karpathy names, transposed to the vault. PR-only workflow gives us a diff to review, but nobody reliably reviews scope-creep on internal vault writes or low-risk skill edits — those just ship. Honest gap: we have no "did this change touch only what was asked?" gate for self-authored edits. The fresh-eyes verify-* critics check quality, not scope-discipline.

3. Think-before-coding / surface assumptions — partially covered, with a real hole. The PRE-DECOMP + IMPLEMENTATION-NOTES requirements (feedback_implementation_notes_sub_agent_pattern) force a dispatched sub-agent to record decisions/deviations/tradeoffs for build scripts, migrations, and financial-execution scripts — that is institutionalized "surface your assumptions." But it only fires on REQUIRED-tier dispatches. IC-mode and single-shot script edits (the majority of small ~/.claude/scripts/ changes) get none of it. The alpaca execution scripts are the sharpest exposure: a silent wrong assumption there moves (paper) money. The paper-trade authorization gate catches deploy intent, not assumption errors inside the script logic.

4. Overcomplication / abstraction bloat — unaudited, plausibly our quiet debt. We have ~30+ Python scripts and a large skill tree; nothing measures whether a 169-line deploy script or a 424-line ingest script is carrying dead code or a bloated construction that should be 100 lines. Karpathy's "umm couldn't you just do this instead?" prompt has no analog in our loop. No skill audits our own code for simplicity; the critics are aimed at PDFs, designs, vault notes, and strategy — not script complexity.

5. The meta-point Karpathy makes — CLAUDE.md instructions don't fully fix behavior — is the one we should take most seriously. He observed these failures persisting despite CLAUDE.md guidance. RDCO's entire bet is a fat CLAUDE.md + fat skills harness. His finding is a caution: prose rules degrade; the things that actually held for him were structural (plan mode, tests-as-gate, declarative loops). Our deterministic audit script is the right shape; our prose feedback memories about scope and simplicity are the shape Karpathy says doesn't reliably stick. Where we have only a memory note and no gate, assume the discipline is aspirational.

Related

• [[2026-05-18-agentway-harness-engineering-claude-code-design-guide]] — the harness-engineering counterpart: Karpathy says prose doesn't fix behavior, agentway says structure (control plane, verification, recovery) does. Read together.
• [[feedback_implementation_notes_sub_agent_pattern]] — our institutionalized "surface assumptions/tradeoffs" for dispatched builds; the partial answer to failure mode #1.
• [[feedback_fresh_eyes_subagent_for_own_artifacts]] — our confirmation-bias guard; relevant to but does NOT currently cover surgical-changes scope discipline (gap noted above).