“What’s an Operational Definition Anyway?” — @CedricChin

Why this is in the vault

This is the foundational piece on how metrics should be defined before they are ever measured — and it maps almost one-for-one onto the row-label layer of Ray Data Co’s MAC (Model Acceptance Criteria) matrix. If MAC is the “what to check” framework, operational definitions are the discipline that makes each check reproducible, comparable across teams, and resistant to silent redefinition. Strong mapping.

The core argument (paraphrased)

Chin opens with a friend’s story: a product team quietly changed “active user” from >$100 spend to >$10 spend — active user counts jumped, and “it was actually a pretty smart move” for hitting quarterly targets. Chin’s horrified response anchors the essay: without recorded metric definitions, every number is rewritable.

Drawing from Donald Wheeler’s Making Sense of Data, he compresses SPC data-collection practice to two rules:

1. “Unless all the values in a set are collected in a consistent manner, the values will not be comparable.”
2. “It is rarely satisfactory to use data collected for one purpose for a different purpose.”

Chin’s added third rule: every metric must come as an operational definition (OD) — three parts:

• Criterion — the Thing you’re measuring (and why)
• Test procedure — how you measure it
• Decision rule — what counts as that Thing

Key moves in the essay:

• Rule 1 application. Can you compare Google Analytics data across a platform switch? Can you compare survey scores when question wording changed? Can you compare office utilisation between buildings using badge systems vs. manual counts? Answer in every case: no. The friend’s team “should’ve deleted all references to the old data” the instant the definition changed.
• Rule 2 application. Wheeler’s plant manager story: productivity = revenue / headcount, but headcount was measured for a different purpose (payroll) and includes the engineering division housed on-site — so his productivity looks artificially low. The fix isn’t better data; it’s tying metric definitions to the purpose of the measurement.
• ODs put “communicable meaning into an adjective.” Words like good, reliable, on-time, active are meaningless until criterion+procedure+rule are agreed between parties. “There can be different operational definitions for the same thing.” It’s not right/wrong; it’s agreement.
• Digital vs. physical measurement. For pure-digital metrics (likes, logins) you can often skip the test procedure because the app instrument is the procedure. For anything physical (office utilisation, manufacturing count, survey) you cannot — and Chin walks through two competing ODs for “employees physically present in office” (manual hand tally vs. wifi MAC address log) to show that neither is “more accurate” — “the number you get will be the result of the method you use.”
• Memorable mnemonic: Thing / Procedure / Rule.
• Closing frame. “The analysis cannot be better than the original data.” Most businesses run on superstition partly because they never invested in the collection layer.

Mapping against Ray Data Co

Mapping strength: strong. This article is functionally the documentation layer that makes the MAC framework defensible.

1. ODs ARE the row-label discipline of the MAC 3×6 matrix. MAC’s 18 cells (scope × basis) describe which category of check applies to a metric. But each cell only becomes executable when the underlying metric has an operational definition — criterion, test procedure, decision rule. Without ODs, a Stop/Pause/Go severity call is undefended: if “active user” can be silently redefined, every threshold is meaningless. MAC without OD discipline is a framework without a foundation. See ../01-projects/data-quality-framework/testing-matrix-template.

2. The agent-deployer’s first job is writing ODs. Per 2026-04-14-levie-agent-deployer-role-jd, the agent-deployer instruments AI workflows and manages evals. Evals are operational definitions for AI output quality — criterion (“factual accuracy on customer-support responses”), test procedure (“LLM-as-judge against gold set X sampled monthly”), decision rule (“response counts as accurate if it matches gold on intent and cites a real policy”). Chin’s framework gives us the template. The agent-deployer skill should probably include an OD authoring step before any dashboard is built.

3. State-ownership requires versioned ODs. The ../04-tooling/rdco-state-ownership-architecture thesis says the client owns the vault + skills + data. Chin’s friend’s story is the dark-mirror version: when ODs aren’t written down, the team can silently rewrite history. For RDCO, the vault must store each client’s ODs with git-style versioning — so when “active user” changes from $100 to $10, the old metric is deprecated, the new one is a separate metric, and dashboard comparisons are broken by design. This is a concrete vault-schema recommendation.

4. phData/MG angle: ODs are the moat against “data-driven theatre.” In consulting engagements (phData-style delivery), the first observable win with a new client is often cleaning up metric definitions — usually 30-60% of dashboards compare values collected inconsistently. Chin’s Rules 1 and 2 give a reusable audit checklist. The MAC engagement opens with an OD audit; the OD audit typically finds more surface-level problems than MAC itself.

5. Harness implication: ODs belong in skills, not prompts. Per the Tan/Levie fat-skills thesis, reusable discipline should live in versioned skill files. An author-operational-definition skill — taking a metric name and walking through criterion/procedure/rule — would be a natural addition to the agent-deployer toolkit. Low effort, high leverage.

Related

• 2026-04-15-commoncog-becoming-data-driven-first-principles — the series capstone; ODs are the collection-layer prerequisite to the XmR/WBR practice that piece describes
• ../01-projects/data-quality-framework/testing-matrix-template — MAC’s 3×6 matrix; ODs are the row-label layer that makes each cell executable
• ../04-tooling/rdco-state-ownership-architecture — versioned ODs are the concrete mechanism behind state-as-moat
• 2026-04-14-levie-agent-deployer-role-jd — OD authoring should be the first step in any agent-deployer engagement
• 2026-04-12-corr-stagnitto-agile-data-warehouse-design-master-synthesis — Corr’s data profiling gives you the raw input for writing ODs; Chin gives you the format
• Forthcoming: author-operational-definition skill — queued on Notion board