06-reference

commoncog goodharts law not useful

Tue Apr 14 2026 20:00:00 GMT-0400 (Eastern Daylight Time) ·reference ·source: Commoncog ·by Cedric Chin

“Goodhart’s Law Isn’t as Useful as You Might Think” — @CedricChin

Why this is in the vault

Chin takes the most-cited meme in data culture (“when a measure becomes a target, it ceases to be a good measure”) and reframes it as a solvable problem via Wheeler/Joiner + Amazon’s WBR. Directly load-bearing for how RDCO defends the MAC framework against the reflexive “but Goodhart’s Law” objection from skeptical clients.

The core argument (paraphrased)

Goodhart’s Law is descriptive, not prescriptive — “pithy and about as practicable as” fortune-cookie wisdom. The useful reformulation comes from Donald Wheeler quoting Brian Joiner: when people are pressured to hit a target, they can (1) improve the system, (2) distort the system, or (3) distort the data.

That reframe generates three concrete solutions: make system-distortion hard, make data-distortion hard, and give people slack to actually improve the system.

Wheeler’s deeper point: “you must listen to the voice of the system” before you can improve it. Targets are the Voice of the Customer; the process itself is the Voice of the Process. “Comparing numbers to specifications will not lead to the improvement of the process.” If all you do is compare current value to target, the only remaining paths are distortion.

Most business processes — unlike weight loss — are processes where “you don’t know the inputs to your desired output.” So step one is discovering the controllable input metrics and their causal relationships to output metrics. “You cannot improve a process by listening to the Voice of the Customer.”

Amazon’s WBR is the canonical implementation:

  1. Split metrics into controllable inputs vs. output metrics. Outputs are lagging, not directly actionable — glanced at. The discussion focuses on exceptions and trends in controllable inputs.
  2. Discover inputs by trial and error. The Fast Track In Stock metric evolved through four iterations (detail pages → page views → in-stock views → in-stock + 2-day-shippable). The WBR itself is the feedback loop that kills wrong inputs.
  3. Build a shared causal model. Weekly cadence over months means the leadership team “builds a causal model of the business in their heads” — and crucially, the same model across the team.
  4. Finance owns the WBR, not the operators. An autonomous group certifies data, chases threads, and has “no skin in the game other than to call it like they see it.” That’s how you make distortion hard.
  5. Audit metrics independently. “Assume that over time something will cause it to drift and skew the numbers.” Separate measurement or customer anecdotes to true up what you report.

Chin’s meta-point: Goodhart’s Law is solvable at the organisational level, but only if you stop treating it as a koan and start treating it as an engineering problem with known solutions from SPC.

Mapping against Ray Data Co

Strong mapping — this is the piece that lets RDCO sell the MAC framework past the “isn’t all measurement self-defeating” objection.

1. MAC’s severity tiers implement “make distortion hard.” The 3×6 testing matrix (column/row/aggregate × absolute/rel-source/rel-production/rel-recon/temporal/human) with Stop/Pause/Go tiers is precisely Joiner’s first two solutions in operational form: catch system-distortion (reconciliation tests, temporal tests) and data-distortion (absolute + relative-to-source tests) before they propagate. See ../01-projects/data-quality-framework/testing-matrix-template.

2. The “controllable input vs. output metric” split is the MAC authoring principle. Clients default to instrumenting output metrics (model accuracy, pipeline uptime) — which are lagging and non-actionable. RDCO’s consulting job is to push them toward controllable inputs (source-system freshness, schema-drift rate, upstream recon deltas) where a weekly cadence can actually change behavior. The Fast-Track-In-Stock evolution story is the template to quote at clients: start rough, iterate, kill dead metrics.

3. State-ownership architecture = the shared causal model, persisted. Amazon’s leadership team builds the causal model in their heads over months of WBR. RDCO’s clients can’t rely on that — turnover, agent handoffs, one-person-consulting reality. The vault + skills architecture is how the causal model persists in artifact form so an agent picks it up next cycle. See ../04-tooling/rdco-state-ownership-architecture.

4. Finance-as-auditor maps to the agent-deployer’s independence posture. Chin’s insight that the WBR works because Finance is structurally independent of the metrics owners is directly analogous to why the agent-deployer role (per 2026-04-14-levie-agent-deployer-role-jd) must sit outside the team whose outputs it audits. For phData vs. MG, this is a differentiator: MG’s “embedded-with-client-team” model conflicts with Chin’s autonomous-auditor principle; phData’s architected-from-outside posture aligns.

5. The “earn the right to criticize” defense, operationalized. When a prospect pushes back with “but Goodhart’s Law,” this article gives a one-page rebuttal: here is the solved version, here is the company that ran it at scale, here is what your objection actually looks like when made rigorous. It converts a conversation-ender into a conversation-starter about which of the three solutions the client is currently missing.

Challenge this article puts on us: Chin assumes the operator can run a weekly cadence against the metrics. For one-person consulting + agent-deployer positioning, the “weekly sync across the leadership team” becomes “weekly sync between the founder and the agent, against the vault.” The MAC drip course needs an explicit cadence component, not just a matrix.