06-reference/research

jev vs custom classifier rdco typed sets

2026-11-02·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
jevcalibrationclassificationtyped-setsbuild-vs-buy

Keep Jev for /ideas because there is no training set, but stop trusting its 0.5 thresholds

The question

"For RDCO's own low-stakes classification use cases (e.g. /ideas type-tagging), would a small custom-trained classifier on existing vault-labeled history actually beat Jev's calibration?"

Raised because [[2026-10-20-technically-whats-a-decision-model]] flagged Jev as "not that well calibrated" and left the universal-pretrained-versus-custom-classifier question explicitly open, while feedback_jev_for_typed_set_problems remains a standing instruction to prefer Jev for typed-set judgments.

What we already know (from the vault)

The label counts, verified by reading the files

This is the load-bearing fact, and it splits the answer in two.

/ideas type-tagging has no training set. 01-projects/personal-brand/ship30/idea-bank.md contains 4 entries carrying both a content type and a 4A angle (the 2026-10-11 manual seed), and 3 later founder entries (2026-10-28) carrying neither. All 4 labeled entries are annotated "Jev: not run (manual seed)," so zero of them are Jev outputs and zero are founder corrections of Jev outputs. The closed type set parsed from idea-types.md has 13 members; the angle set has 4. That is 4 labeled examples against 13 type classes and 52 joint type-angle cells, and all 4 labeled examples sit in a single angle (Analytical). No held-out split is constructible. A custom classifier is not a worse option here, it is not an option.

The rest of the vault is label-rich, and none of it is on Jev. Counted directly from frontmatter:

Label field Labeled notes Classes Shape
sponsored: in 06-reference 1,901 clean (1,033 false / 866 true) 2 near-balanced binary
newsletter_format: in 06-reference 1,628 26 observed, top 4 = 90.6% long-tailed multiclass
category: in 06-reference/research 461 usable question-plus-category pairs of 467 briefs 7 canonical after case-folding imbalanced multiclass (Strategy 48.8%)

So the premise of the question inverts on contact with the data. The one job Jev is wired into is the one job that cannot be custom-trained, and the three jobs with enough labels to train and to measure calibration are not using Jev at all.

One verified defect found while checking this. jev_judge.py's idea-bank parser expects headlines of the form - **[I-NNN]** ... "headline". The current idea-bank.md uses - **2026-10-11-01** · ... instead. Importing the module and calling load_dedup_corpus() returns 40 headlines: 31 from topic-bank-v1, 6 from week1-drafts, 3 from drafts/, and 0 from the idea bank. /ideas therefore cannot currently detect that a new idea duplicates a previously banked idea. This is a code defect independent of the Jev question, but it also closes off the path by which the /ideas label pool would otherwise grow.

What the web says

Convergences and contradictions

Synthesis for RDCO

Decision: hybrid, staged on label count, and the first stage is not a model change at all. Keep Jev as the judgment layer for /ideas, because with 4 labeled examples against 13 classes there is no alternative to keep it honest against. But stop treating Jev's returned probabilities as calibrated probabilities, because the evidence says they are not calibrated off-distribution and RDCO is entirely off-distribution. Concretely, jev_judge.py hardcodes DUP_FLAG_P = 0.5, RELATED_P = 0.5 and TYPE_CONF_REVIEW = 0.5. Each of those is a bet that a Jev probability of 0.5 corresponds to a 50% real-world frequency. At an off-distribution expected calibration error near 0.107, that bet is wrong by roughly a fifth of the distance to the threshold, in an unknown direction. The cheap fix is not to swap models: it is to use Jev's output as an ordinal rank rather than a probability until enough labels exist to fit a calibrator, and to start accruing those labels now by logging every Jev judgment alongside the value a human finally kept. That logging does not exist today, which is why this question has stayed unresolved through six-plus vault notes - nobody is writing down the ground truth.

The sharper strategic finding is that the standing instruction has been aimed at the wrong use case. feedback_jev_for_typed_set_problems sent Jev at /ideas, which is RDCO's lowest-volume, highest-class-count, zero-label typed-set job - the worst possible place to evaluate it and the worst place to get value from it. Meanwhile sponsored: is a near-balanced binary with 1,901 labels on disk, newsletter_format: has 1,628, and research-brief category: has 461 clean input-label pairs, and all three are hand-adjudicated today with no decision model anywhere in the loop. If the point of preferring Jev is to replace prompt-and-parse with composable typed judgments, the three jobs where that would pay are untouched. The 26 observed values of newsletter_format: against a top-4 that covers 90.6% is itself a drift symptom that feedback_founder_shared_filing_contract already tries to fix by instruction ("never invent newsletter_format"); a closed-set classifier with a no-match outcome is the mechanical version of that instruction, which is strictly more reliable than telling an agent not to improvise.

The switch-over threshold, in concrete terms. Do not frame this as Jev-versus-custom. Frame it as which layer sits on top of Jev, promoted by labels per class, where a label counts only if a human set or confirmed the final value - never Jev's own output, or the model trains on its own predictions:

Accuracy versus calibration, and the metric that would actually settle it. Accuracy is the fraction of top-1 answers that are right; it discards the confidence entirely, so a model that says 0.51 and a model that says 0.99 score identically. Calibration asks whether a stated probability matches the observed frequency at that probability, and it is the property that makes a judgment composable in code, because every threshold in jev_judge.py is an arithmetic statement about a probability. The vault's prior Jev evidence is accuracy-only and therefore cannot settle this. The settling metric is multiclass Brier score on a pre-registered, class-stratified held-out split, reported with N and against two baselines: uniform (1/K) and always-base-rate. Brier is a proper scoring rule and decomposes into a calibration term and a refinement term, so it cannot be gamed by hedging and it does not depend on a binning choice. Expected calibration error is the secondary diagnostic, reported against a bootstrap noise floor, and demoted from primary precisely because the UCLA audit showed it moving 0.071 on question phrasing alone. The reliability curve is the third artifact, because it is the only one that shows where the miscalibration lives rather than how much there is. Accuracy gets reported too, but as context, not as the gate - per [[brier-score]]'s own rule against using accuracy on probabilistic problems.

The cheapest experiment that would decide it

Everything below runs on this machine today, on data already on disk, using the TypeSafe key the /ideas path already reads from 1Password at call time. No new labeling and no new credentials.

Target: research-brief category routing. Not /ideas - /ideas has 4 labels and no held-out set is constructible, so no experiment there can produce a number. The category task is the same shape as /ideas type-tagging (short text in, exactly one of N RDCO-invented classes out, imbalanced, no public analogue, bespoke criteria), so the calibration result transfers, and it is the only RDCO typed-set job with enough labels to measure calibration at all while still being small enough to finish in one sitting.

Setup. Case-fold category: (folding strategy into Strategy, content into Content), drop the 3 singleton junk values (Strategy / Tooling, strategy-trends, enterprise-data-architecture), leaving n=458 across 7 classes. Input is the question: field only. Write the class-stratified 50/50 split to a file and freeze it before any arm runs.

Arms, all scored on the identical held-out half.

Pre-registered success bar, written down before the run.

  1. Jev stays the default unless arm B or arm C beats arm A's held-out multiclass Brier by at least 0.02 absolute and a 1,000-resample bootstrap 95% confidence interval on the paired difference excludes zero. A win on accuracy alone does not count.
  2. Independent of which arm wins: if arm A's expected calibration error exceeds 3x the bootstrap noise floor, the three 0.5 thresholds in jev_judge.py are declared unvalidated and must be refit by Platt scaling or replaced with rank-based logic. This is the check that matters most for /ideas, because it is the one whose conclusion transfers to a task with no training data.
  3. If arm C wins, stage 2 becomes the standing architecture for every RDCO typed-set job that clears 300 confirmed labels, and sponsored: is the next target.
  4. Reporting discipline: N, both baselines, and the reliability curve ship with every number, or the result does not count. No arm's figure is quoted without its held-out N.

Scale. About 458 Jev calls across one Choice question. For reference on order of magnitude, the independently-run pre-registered study cited above reported 5,721 Jev calls costing $0.176 at TypeSafe's published list price, so this run is roughly a penny of metered spend and sits far inside autonomous authority per project_rdco_spending_authority. Arms B and C are local CPU work.

Not part of the experiment, but do it first because it is a correctness fix: repair the load_dedup_corpus() idea-bank regex so the 7 banked ideas actually enter the dedup set, and add the human-confirmed-label log that stages 1 through 3 depend on.

Why this is in the vault

This brief resolves the specific open question feedback_jev_for_typed_set_problems has been carrying unexamined since 2026-10-11 and that [[2026-10-20-technically-whats-a-decision-model]] named as unresolved: it establishes by direct file inspection that /ideas has 4 labeled examples against 13 classes (so the custom-classifier option does not exist there), that three other vault label pools of 461 to 1,901 examples do clear the training bar and are not on Jev, and that the vault's existing Jev evidence in [[2026-09-27-alphasignal-system-one-models-jev-laya-clm8b]] is accuracy-only and must stop being cited on calibration. It also supplies the pre-registered test and threshold staircase that turn the standing instruction from a preference into something measurable.

Open follow-ups

Related

Sources

Vault

Primary artifacts inspected directly

Web