06-reference

commoncog how to become data driven

Tue Apr 14 2026 20:00:00 GMT-0400 (Eastern Daylight Time) ·reference ·source: Commoncog ·by Cedric Chin

“How to Become Data Driven” — @CedricChin

Why this is in the vault

Part 2 of Chin’s Becoming Data Driven series. This is the essay that names the single gating skill for data-driven-ness — telling routine from exceptional variation — and then walks all the way from Wheeler’s shoe-company parable through the XmR chart construction recipe, predictable-vs-unpredictable process taxonomy, and the percentage-metrics trap that makes most monthly management reports actively misleading. This is the operator-literacy prerequisite that the Part 12 cornerstone and the entire MAC framework assume. It is also the clearest single-essay briefing we have on how to staff and train the agent-deployer role at phData/MG scale.

The core argument (paraphrased)

Chin opens with Wheeler’s shoe-company president who kept a daily defects chart on his wall and, when asked how the plant was doing, replied “some days are better than others!” Chin’s wager: most operators have exactly this relationship with Google Analytics, MQL dashboards, and Stripe graphs — watch the line wobble, shrug at the trend, never know what to do with the information.

Voice of the Process vs. Voice of the Customer. Quoting Wheeler: “Before you can improve any system you must listen to the voice of the system.” The Voice of the Customer is targets, specs, budgets, KPIs. The Voice of the Process is what the underlying system actually produces. “The specification approach does not reveal any insights into how the process works.” Comparing outputs to targets creates only three options: distort the system, distort the data, or (rarely) improve the system — and distortion is always the path of least resistance.

The three questions. Any time series is the output of some process. Ask:

  1. Did you successfully change the process? Did the intervention produce a permanent shift in the mean?
  2. Did something else change? Exogenous shock — celebrity link, partner policy change, machine blew up last July.
  3. Is this just routine variation? Should you ignore it entirely, rather than author a fabricated explanation for why MQLs dipped?

The compressed thesis. “Telling the difference between routine versus exceptional variation is the beginning of becoming data driven.” Stronger: “it is impossible to be properly data driven if you do not fully understand variation.” You can’t investigate real change if you can’t distinguish signal from noise, and you’ll burn the team out investigating every “number go down.” Deming’s payoff: “every instance of exceptional variation is an opportunity for process improvement” — positive spike, find what worked; negative dip, find the cause before it compounds.

Process behaviour charts (XmR). Wheeler prefers “process behaviour chart” over “process control chart” — you’re reading time series to understand process nature, which you don’t necessarily control. The XmR pair: an X-chart of the metric plus a Moving Range chart of absolute successive differences. Three lines matter: Upper Natural Process Limit, Lower Natural Process Limit, Upper Range Limit. Point outside the limits = exceptional variation = investigate. Point within = routine = usually noise.

The construction recipe, which is surprisingly small:

The scaling constants approximate three-sigma limits. Chin: “Is it really that simple? Yes, it’s that simple.” Wheeler’s three-sigma choice is deliberately conservative — wide enough to filter routine variation and suppress false alarms across essentially any distribution shape, narrow enough to still catch economically significant signals. You only need five to six data points to start, which means you can re-plot quickly after an intervention to see whether the limits have genuinely shifted.

Predictable vs. unpredictable processes. The big conceptual payoff:

This inverts folk wisdom (“if it ain’t broke, don’t fix it”; “the squeaky wheel gets the grease”). SPC says: reengineer the good processes, tweak the bad ones. The marketing channel you’re proud of at 15% conversion with marginal tweaks each quarter? If it’s predictable, you will never beat 15% by tweaking — you have to rethink the approach. Conversely, the “badly performing” process you’re about to tear down might just be unpredictable — find and remove the special causes first.

Chin also notes: “management is prediction.” You cannot run a business well if you cannot predict the outcomes of your actions (predictable within limits, not omniscient). An unpredictable process literally cannot be managed — you’re always being surprised by the system you nominally run.

A worked example: the monthly report. A manufacturing facility’s July report shows first-time approvals down 23%, in-process inventory up 42%, on-time closings down 21.8%. Management instinct: haul in the three department heads and demand explanations — especially for the 42% WIP spike, since work-in-progress is a classic bottleneck signal. But the production manager pulls three years of WIP data and plots an XmR chart. The 42% jump is inside the Natural Process Limits. It’s routine variation. There is nothing to investigate. Worse: if the manager decreed WIP must stay within ±20% of mean, the only possible outcomes are workers distorting the system or distorting the data, because the process has never behaved within those limits. Wheeler: “Such decrees, by themselves, do nothing to change or improve the system.”

Four lessons from this single example:

  1. Listening to the Voice of the Process means first understanding the process’s actual current variation. Only then can you reason about improvement.
  2. SPC only works on processes performed with consistency. Ad-hoc activities (early-market startup sales, one-off deals) produce XmR charts that reveal nothing.
  3. If a process shows only routine variation, incremental pressure will never change its behaviour — a fundamental redesign is required.
  4. Percentage metrics are dangerous.

The percentage-metrics trap. Three reasons percent-change columns in monthly reports are systematically misleading:

The sequel example: on-time shipment looks innocent — 91.0%, only 0.3% below average, 0.9% below last July. But the XmR chart shows six months outside the Natural Process Limits across three years. “You have already missed three opportunities to improve this process.” The process had been signaling a recurring problem for years; the percent-change monthly report format rendered it invisible. Wheeler: “How many more signals are you going to miss before this problem causes you to lose a large account?”

The closing frame. Chin: understanding variation is the beginning of data literacy the way reading is the beginning of literacy — profound because a whole universe opens once you have it, obvious once you do. “Those who don’t will stick out like illiterate sore thumbs.”

Appendix worth keeping. Wheeler’s five uses of process behaviour charts:

  1. As a report card.
  2. As feedback for process improvement.
  3. As feedback for short, simple experiments on a process.
  4. As extended monitoring — running multiple XmR charts over time to discover which metric is the best leading indicator of product/process performance. Chin flags this as structurally identical to Amazon’s WBR metric discovery loop.
  5. As the on-ramp to continuous improvement. XmR charts don’t arrive in an org that already has continuous improvement; they are how an org gets to continuous improvement, by propagating a particular style of thinking.

Edge cases boil down to: plot separate XmR charts for logically separated phenomena. Post-intervention data goes in a new chart. Seasonal data (weekend vs. weekday footfall) gets split.

Mapping against Ray Data Co

Mapping strength: strong. This essay is the operator-literacy kernel under essentially everything RDCO builds.

1. This is the literacy prerequisite for MAC. The Part 12 note flagged that MAC only works if operators have enough statistical-thinking literacy to not treat every spike as a crisis. This essay is that literacy, compressed to its minimum viable kernel: three questions against any time series, XmR construction in five steps, predictable-vs-unpredictable taxonomy. Any MAC drip course or coaching engagement should open with a version of this content before the 3×6 matrix ever appears — otherwise clients will escalate every Pause-tier signal to Stop-tier, burn out the on-call rotation, and conclude MAC doesn’t work. See ../01-projects/data-quality-framework/testing-matrix-template.

2. Voice of the Process vs. Voice of the Customer is the agent-deployer’s default failure mode, stated cleanly. Per 2026-04-14-levie-agent-deployer-role-jd, most enterprises instrument AI outputs against stated specs — accuracy %, SLA targets, eval pass rates — without ever building a model of the underlying agent-process that produces them. Chin’s warning applies exactly: comparing outputs to spec numbers never teaches you how the agent behaves. The agent-deployer exists to listen to the Voice of the Process — eval traces, MAC cell movements, tool-call distributions — not just the Voice of the acceptance target. The phData vs. MG decision partly comes down to which vendor culturally defaults to process-listening over spec-checking; worth explicitly probing in vendor conversations.

3. Predictable-vs-unpredictable maps directly onto MAC severity response. Chin’s taxonomy — reengineer the good processes, tweak the bad ones — is the operational logic MAC’s Stop/Pause/Go tiers imply but don’t spell out. A client model running cleanly within MAC limits for six months is a predictable process; incremental MAC-cell tightening will yield nothing, and the only real lever is a model redesign or ingest-source change. A model throwing Pause-tier signals weekly is unpredictable; first move is removing the special-cause noise, not adjusting thresholds. This should be explicit coaching material.

4. The percentage-metrics trap is the exact failure mode of most AI eval dashboards. “Accuracy up 3% week-over-week” is the AI-era “on-time shipments at 91%.” The percent-change framing systematically hides the XmR signal underneath. RDCO’s MAC dashboards should plot absolute values with UNPL/LNPL lines, not percent-deltas. Worth baking into the state-ownership architecture: the vault stores raw eval runs, the skills compute XmR limits, and the surfaced dashboard is process-behaviour-shaped by default. See ../04-tooling/rdco-state-ownership-architecture.

5. The agent-deployer is the modern SPC operator; MAC is the WBR-for-agents. Chin’s five uses of XmR charts translate almost unchanged into the agent-deployer job description: report card for stakeholders, feedback for prompt/tool iterations, feedback for small A/B experiments on agent behaviour, extended monitoring to find which eval metric is the best leading indicator of downstream quality, and on-ramp to continuous improvement discipline. That fifth use is the real RDCO consulting product: not handing a client a MAC skill and leaving, but installing the style of thinking that XmR charts propagate when operators use them for months.

One thing this article specifically sharpens: Chin is explicit that SPC only works on consistently-performed activities. That is a live constraint for RDCO’s consulting pitch — early-stage AI agent deployments where prompts, tools, and scope are all churning week-to-week do not yet have a “process” in the SPC sense. MAC lands cleanly on mature, high-volume, stable-scope agent workloads (the phData/MG enterprise shape), not on founder-mode prototypes. The go-to-market should probably lead with stable deployments and treat early-stage instrumentation as a different (lighter) engagement.