Keep Jev for /ideas because there is no training set, but stop trusting its 0.5 thresholds
The question
"For RDCO's own low-stakes classification use cases (e.g. /ideas type-tagging), would a small custom-trained classifier on existing vault-labeled history actually beat Jev's calibration?"
Raised because [[2026-10-20-technically-whats-a-decision-model]] flagged Jev as "not that well calibrated" and left the universal-pretrained-versus-custom-classifier question explicitly open, while feedback_jev_for_typed_set_problems remains a standing instruction to prefer Jev for typed-set judgments.
What we already know (from the vault)
- The standing instruction has one implementation, and it is /ideas. A grep across all 62 installed skills finds Jev referenced in exactly three files, all under
~/.claude/skills/ideas/. Nothing else in RDCO calls Jev. The instruction is broad; the production surface is one skill. - The vault's own prior benchmark measured the wrong thing for this question. [[2026-09-27-alphasignal-system-one-models-jev-laya-clm8b]] records CLM-8B beating Jev 81.6% versus 71.1% and 87.6% versus 83.1% on judge-pattern tasks. Those are accuracy figures on 38 and 30 tasks respectively. They say nothing about calibration, which is why that note left the question unresolved rather than answering it.
- RDCO already has a literature-backed label-volume threshold, from a different problem. [[2026-05-19-cai-critic-graduation-per-axis-threshold]] converged on ~100-200 human labels per axis to reach 85% agreement, plus a separate held-out reliability check, and flagged that single-metric thresholds are gameable. That brief was about critic graduation, not classifier training, but the per-class arithmetic transfers.
- RDCO already has the metric doctrine written down. [[brier-score]] states the rule this question turns on: "Accuracy ignores confidence entirely... Never use accuracy on probabilistic problems." It also specifies the discipline that most calibration claims skip - always report N, and always report the majority-class and always-0.5 baselines alongside the score.
- The vendor itself declines to publish a calibration number and pushes validation onto the user. The installed
typesafe:typesafe-aiskill says "System One models are trained for calibrated decisions; validate their performance in the target domain," warns that "Choice/Score confidence summarizes distribution concentration, not overall workflow correctness or permission to act," and tells the reader to "treat cookbook thresholds and demo results as examples to evaluate, not universal rules." It carries no expected-calibration-error or Brier figure anywhere. It also, notably, endorses the hybrid path outright: "With labeled outcomes, those signals can become classical ML features."
The label counts, verified by reading the files
This is the load-bearing fact, and it splits the answer in two.
/ideas type-tagging has no training set. 01-projects/personal-brand/ship30/idea-bank.md contains 4 entries carrying both a content type and a 4A angle (the 2026-10-11 manual seed), and 3 later founder entries (2026-10-28) carrying neither. All 4 labeled entries are annotated "Jev: not run (manual seed)," so zero of them are Jev outputs and zero are founder corrections of Jev outputs. The closed type set parsed from idea-types.md has 13 members; the angle set has 4. That is 4 labeled examples against 13 type classes and 52 joint type-angle cells, and all 4 labeled examples sit in a single angle (Analytical). No held-out split is constructible. A custom classifier is not a worse option here, it is not an option.
The rest of the vault is label-rich, and none of it is on Jev. Counted directly from frontmatter:
| Label field | Labeled notes | Classes | Shape |
|---|---|---|---|
sponsored: in 06-reference |
1,901 clean (1,033 false / 866 true) | 2 | near-balanced binary |
newsletter_format: in 06-reference |
1,628 | 26 observed, top 4 = 90.6% | long-tailed multiclass |
category: in 06-reference/research |
461 usable question-plus-category pairs of 467 briefs | 7 canonical after case-folding | imbalanced multiclass (Strategy 48.8%) |
So the premise of the question inverts on contact with the data. The one job Jev is wired into is the one job that cannot be custom-trained, and the three jobs with enough labels to train and to measure calibration are not using Jev at all.
One verified defect found while checking this. jev_judge.py's idea-bank parser expects headlines of the form - **[I-NNN]** ... "headline". The current idea-bank.md uses - **2026-10-11-01** · ... instead. Importing the module and calling load_dedup_corpus() returns 40 headlines: 31 from topic-bank-v1, 6 from week1-drafts, 3 from drafts/, and 0 from the idea bank. /ideas therefore cannot currently detect that a new idea duplicates a previously banked idea. This is a code defect independent of the Jev question, but it also closes off the path by which the /ideas label pool would otherwise grow.
What the web says
- No rigorous same-dataset head-to-head of Jev against a small supervised classifier on calibration exists. Stating that flatly, as instructed. Every comparison found is a single run, a different dataset, or an unverified community report. One review asserts no independent Jev benchmark existed as of September 2026 at all.
- Jev's calibration degrades sharply off-distribution, and that is the regime RDCO operates in. An independent study reports expected calibration error of 0.107 on 900 synthetic support tickets, described as 4.4 times the measured noise floor of 0.024, against 0.024-0.032 on public benchmarks (beri.net digest). RDCO's label sets are entirely bespoke, with no public analogue, which is the off-distribution case.
- Question decomposition is a far larger effect than model choice. On a 2,000-email phishing benchmark, Jev asked a single "is this phishing?" question scored 62.6% accuracy with calibration error 0.154. The same model asked five narrow signals in one call, combined by a logistic regression fitted on 1,000 labeled emails, scored 95.0%. That is roughly 32 accuracy points from restructuring the question rather than changing the model. The same source reports the decomposition helped Jev specifically and did not help Claude Haiku, whose best single signal beat its own composite.
- A small fine-tuned model does beat Jev on calibration in the one case anyone reports, and the case is weak evidence. A community commenter reports Qwen3-4B with LoRA on 1,000 of the phishing emails reaching 97.4% accuracy and calibration error 0.010 on a disjoint 500. Unverified, single run, not a controlled comparison. Treat as a direction, not a number.
- Vendor-published figures, labeled as such. TypeSafe publishes 193.6x faster and 444.6x cheaper against an average of two frontier models, $0.042 per million input tokens with free output, and 70-500ms latency. The same digest notes TypeSafe's charts show a "0%" hallucination rate that "is not empirical," and that TypeSafe's launch post publishes no calibration metric. A pentest preprint cites expected calibration error of 0.246 for Jev on TypeSafe's own evaluation set while warning the figure is not comparable across tasks.
- Calibration measurement is itself fragile, which argues against making expected calibration error the primary metric. A UCLA audit of Jev 1.13.0 across 3,244 evaluations of 2,372 texts found calibration error moving by 0.071 between two phrasings of the same question. A metric that swings that much on framing cannot be the sole gate.
- The academic crossover for supervised-beats-zero-shot is low, and it moves against Jev as class count rises. A 32-dataset, 8-language study found few-shot fine-tuning with ~10 examples per label beat zero-shot classification on most tasks (arXiv:2502.11830). A separate study found fine-tuned small models significantly outperforming zero-shot generative models on all its case studies, with performance saturating after roughly 200 labels (arXiv:2406.08660). Both report zero-shot holding up best when the class count is small, with fine-tuning pulling ahead as categories multiply. /ideas type-tagging has 13 classes, which is the unfavorable end of that finding.
Convergences and contradictions
- Convergence on the real lever. Schneider's "pick two of universal, fast, cheap" framing in [[2026-10-20-technically-whats-a-decision-model]], TypeSafe's own instruction to "ask one narrow, coherent judgment per question" and "split independently useful dimensions," and the phishing 62.6%-to-95.0% result all point the same way: the win comes from decomposing the judgment and fitting a thin layer on top, not from picking a different model.
jev_judge.pyalready does a weak version of this (the same type question asked twice with reversed option order, the two distributions averaged, disagreement raised as a review signal). The upgrade is continuous with what exists. - Contradiction on label volume, and it matters for the threshold. [[2026-05-19-cai-critic-graduation-per-axis-threshold]] says 100-200 per axis; the text-classification literature says ~10 per class to beat zero-shot and ~200 total to saturate. These are not actually in conflict - the CAI figure is for an LLM-as-judge reaching 85% agreement with a human on a fuzzy quality axis, which is a harder bar than a classifier beating zero-shot on a closed set. The practical reading is a staircase rather than a single cutover, which is how the recommendation below is built.
- Contradiction between the vault's prior benchmark and this question. [[2026-09-27-alphasignal-system-one-models-jev-laya-clm8b]] has been carried as the vault's build-versus-buy evidence on Jev. It is accuracy-only. It should not be cited again as bearing on calibration.
Synthesis for RDCO
Decision: hybrid, staged on label count, and the first stage is not a model change at all. Keep Jev as the judgment layer for /ideas, because with 4 labeled examples against 13 classes there is no alternative to keep it honest against. But stop treating Jev's returned probabilities as calibrated probabilities, because the evidence says they are not calibrated off-distribution and RDCO is entirely off-distribution. Concretely, jev_judge.py hardcodes DUP_FLAG_P = 0.5, RELATED_P = 0.5 and TYPE_CONF_REVIEW = 0.5. Each of those is a bet that a Jev probability of 0.5 corresponds to a 50% real-world frequency. At an off-distribution expected calibration error near 0.107, that bet is wrong by roughly a fifth of the distance to the threshold, in an unknown direction. The cheap fix is not to swap models: it is to use Jev's output as an ordinal rank rather than a probability until enough labels exist to fit a calibrator, and to start accruing those labels now by logging every Jev judgment alongside the value a human finally kept. That logging does not exist today, which is why this question has stayed unresolved through six-plus vault notes - nobody is writing down the ground truth.
The sharper strategic finding is that the standing instruction has been aimed at the wrong use case. feedback_jev_for_typed_set_problems sent Jev at /ideas, which is RDCO's lowest-volume, highest-class-count, zero-label typed-set job - the worst possible place to evaluate it and the worst place to get value from it. Meanwhile sponsored: is a near-balanced binary with 1,901 labels on disk, newsletter_format: has 1,628, and research-brief category: has 461 clean input-label pairs, and all three are hand-adjudicated today with no decision model anywhere in the loop. If the point of preferring Jev is to replace prompt-and-parse with composable typed judgments, the three jobs where that would pay are untouched. The 26 observed values of newsletter_format: against a top-4 that covers 90.6% is itself a drift symptom that feedback_founder_shared_filing_contract already tries to fix by instruction ("never invent newsletter_format"); a closed-set classifier with a no-match outcome is the mechanical version of that instruction, which is strictly more reliable than telling an agent not to improvise.
The switch-over threshold, in concrete terms. Do not frame this as Jev-versus-custom. Frame it as which layer sits on top of Jev, promoted by labels per class, where a label counts only if a human set or confirmed the final value - never Jev's own output, or the model trains on its own predictions:
- Stage 0, fewer than 10 confirmed labels per class. Jev raw. Consume the ranking, not the number. No threshold in code may be justified by "P is above 0.5." /ideas is here, at 4 labels total, and will stay here for a long time at current volume.
- Stage 1, at least 10 per class and at least 30 total for that question. Fit a one-parameter calibrator (Platt or temperature scaling) per question on Jev's output, on half the labels, scored on the other half. One parameter, no feature engineering, and it directly repairs the 0.5 thresholds.
- Stage 2, at least 50 per class and at least 300 total. Decompose the question into 3-6 narrow Noul signals in a single call and fit a multinomial logistic regression over Jev's probability vector. This is the phishing recipe and the largest measured effect in the evidence. Research-brief
category:is at this stage now for its top three classes. - Stage 3, at least 100-200 per class, class-balanced. Train a standalone small supervised classifier (sentence embeddings plus logistic regression, or SetFit) and retire the Jev call for that question only if it wins on held-out Brier and on expected calibration error and costs less to run.
sponsored:and the top fournewsletter_format:classes already clear this bar; nothing in /ideas is within two orders of magnitude of it.
Accuracy versus calibration, and the metric that would actually settle it. Accuracy is the fraction of top-1 answers that are right; it discards the confidence entirely, so a model that says 0.51 and a model that says 0.99 score identically. Calibration asks whether a stated probability matches the observed frequency at that probability, and it is the property that makes a judgment composable in code, because every threshold in jev_judge.py is an arithmetic statement about a probability. The vault's prior Jev evidence is accuracy-only and therefore cannot settle this. The settling metric is multiclass Brier score on a pre-registered, class-stratified held-out split, reported with N and against two baselines: uniform (1/K) and always-base-rate. Brier is a proper scoring rule and decomposes into a calibration term and a refinement term, so it cannot be gamed by hedging and it does not depend on a binning choice. Expected calibration error is the secondary diagnostic, reported against a bootstrap noise floor, and demoted from primary precisely because the UCLA audit showed it moving 0.071 on question phrasing alone. The reliability curve is the third artifact, because it is the only one that shows where the miscalibration lives rather than how much there is. Accuracy gets reported too, but as context, not as the gate - per [[brier-score]]'s own rule against using accuracy on probabilistic problems.
The cheapest experiment that would decide it
Everything below runs on this machine today, on data already on disk, using the TypeSafe key the /ideas path already reads from 1Password at call time. No new labeling and no new credentials.
Target: research-brief category routing. Not /ideas - /ideas has 4 labels and no held-out set is constructible, so no experiment there can produce a number. The category task is the same shape as /ideas type-tagging (short text in, exactly one of N RDCO-invented classes out, imbalanced, no public analogue, bespoke criteria), so the calibration result transfers, and it is the only RDCO typed-set job with enough labels to measure calibration at all while still being small enough to finish in one sitting.
Setup. Case-fold category: (folding strategy into Strategy, content into Content), drop the 3 singleton junk values (Strategy / Tooling, strategy-trends, enterprise-data-architecture), leaving n=458 across 7 classes. Input is the question: field only. Write the class-stratified 50/50 split to a file and freeze it before any arm runs.
Arms, all scored on the identical held-out half.
- A — Jev zero-shot. One Choice question over the 7 criteria, using /curiosity's actual category definitions, pinned to an explicit model version rather than
jev-latest, with a no-match option included per TypeSafe's own guidance. - B — Custom only. Local sentence embeddings or TF-IDF, plus multinomial logistic regression, fitted on the fit half. Scikit-learn on CPU, seconds.
- C — Hybrid. Jev's 7-probability vector from arm A as features, plus logistic regression fitted on the fit half. This is the stage-2 recipe and the arm the evidence predicts will win.
- Baselines. Uniform (1/7) and always-base-rate, so a result that merely encodes the 48.8% Strategy prior is visible as such.
Pre-registered success bar, written down before the run.
- Jev stays the default unless arm B or arm C beats arm A's held-out multiclass Brier by at least 0.02 absolute and a 1,000-resample bootstrap 95% confidence interval on the paired difference excludes zero. A win on accuracy alone does not count.
- Independent of which arm wins: if arm A's expected calibration error exceeds 3x the bootstrap noise floor, the three 0.5 thresholds in
jev_judge.pyare declared unvalidated and must be refit by Platt scaling or replaced with rank-based logic. This is the check that matters most for /ideas, because it is the one whose conclusion transfers to a task with no training data. - If arm C wins, stage 2 becomes the standing architecture for every RDCO typed-set job that clears 300 confirmed labels, and
sponsored:is the next target. - Reporting discipline: N, both baselines, and the reliability curve ship with every number, or the result does not count. No arm's figure is quoted without its held-out N.
Scale. About 458 Jev calls across one Choice question. For reference on order of magnitude, the independently-run pre-registered study cited above reported 5,721 Jev calls costing $0.176 at TypeSafe's published list price, so this run is roughly a penny of metered spend and sits far inside autonomous authority per project_rdco_spending_authority. Arms B and C are local CPU work.
Not part of the experiment, but do it first because it is a correctness fix: repair the load_dedup_corpus() idea-bank regex so the 7 banked ideas actually enter the dedup set, and add the human-confirmed-label log that stages 1 through 3 depend on.
Why this is in the vault
This brief resolves the specific open question feedback_jev_for_typed_set_problems has been carrying unexamined since 2026-10-11 and that [[2026-10-20-technically-whats-a-decision-model]] named as unresolved: it establishes by direct file inspection that /ideas has 4 labeled examples against 13 classes (so the custom-classifier option does not exist there), that three other vault label pools of 461 to 1,901 examples do clear the training bar and are not on Jev, and that the vault's existing Jev evidence in [[2026-09-27-alphasignal-system-one-models-jev-laya-clm8b]] is accuracy-only and must stop being cited on calibration. It also supplies the pre-registered test and threshold staircase that turn the standing instruction from a preference into something measurable.
Open follow-ups
- Does arm C (Jev probabilities as features into a logistic regression) generalize from the 7-class category task to the 26-value
newsletter_format:long tail, or does the long tail need a hierarchical decomposition with a no-match outcome instead? sponsored:has 1,901 labels but the source newsletter bodies are not retained in the vault, and the note's own "⚠️ Sponsorship" section leaks the label. Is there a leakage-free input for that task, or is the 1,901 figure unusable as a training set?- What is the actual per-class confirmed-label accrual rate for /ideas once override logging exists? At current volume /ideas may never reach stage 1, which would make the Jev dependency permanent rather than transitional, and that changes how much the uncalibrated thresholds matter.
- Should the stage-3 bar adopt Cohen's kappa alongside Brier, given [[2026-05-19-cai-critic-graduation-per-axis-threshold]]'s finding that single metrics are gameable and that RDCO's labels come from one labeler with no inter-annotator agreement measurable at all?
- Cloudflare's Clef (classification) and Clairic (regression) per
reference_cloudflare_clef_clairicare a fourth arm nobody has priced or benchmarked. Do either publish a calibration metric, which would make them the first vendor in this category to do so? - The UCLA audit's finding that calibration error moves 0.071 on question phrasing suggests prompt-phrasing variance should itself be measured as an arm. Is there a cheap phrasing-robustness check worth standardizing across all RDCO Jev calls?
Related
- [[2026-10-20-technically-whats-a-decision-model]] — the note that raised this question and flagged Jev as "not that well calibrated"; this brief answers its open universal-versus-custom question for RDCO's actual jobs
- [[2026-09-27-alphasignal-system-one-models-jev-laya-clm8b]] — the vault's prior Jev-versus-CLM-8B-versus-Laya benchmark; this brief establishes it is accuracy-only and does not bear on calibration
- [[2026-09-15-every-typesafe-jev-vibe-check]] — opened the Jev due-diligence thread and landed on "cheap pre-filter, not a critic replacement," which the staircase here makes concrete
- [[2026-05-19-cai-critic-graduation-per-axis-threshold]] — the vault's prior label-volume threshold work (~100-200 per axis); reused here as the stage-3 bar and contrasted against the text-classification literature's lower crossover
- [[brier-score]] — supplies the metric doctrine this brief applies: proper scoring rule, always report N and both baselines, never use accuracy on probabilistic problems
- [[feedback_jev_for_typed_set_problems]] — the standing instruction under test; this brief keeps it but narrows it and attaches a measurement bar
- [[reference_cloudflare_clef_clairic]] — the rival primitives (Clef, Clairic) left unpriced and unbenchmarked by this brief
Sources
Vault
~/rdco-vault/06-reference/2026-10-20-technically-whats-a-decision-model.md~/rdco-vault/06-reference/2026-09-27-alphasignal-system-one-models-jev-laya-clm8b.md~/rdco-vault/06-reference/research/2026-05-19-cai-critic-graduation-per-axis-threshold.md~/rdco-vault/06-reference/concepts/brier-score.md~/rdco-vault/01-projects/personal-brand/ship30/idea-bank.md,idea-types.md,2026-10-05-topic-bank-v1.md,posted-log.md
Primary artifacts inspected directly
~/.claude/skills/ideas/jev_judge.pyand~/.claude/skills/ideas/SKILL.md— current production path; thresholds and the dedup-corpus defect verified by importing the module~/.claude/plugins/cache/typesafe-ai/typesafe/0.5.7/skills/typesafe-ai/SKILL.md— installed vendor skill; source of the vendor's own "validate in the target domain" language and the absence of any published calibration metric- Frontmatter label counts computed over
~/rdco-vault/06-reference/*.md(2,571 notes) and~/rdco-vault/06-reference/research/*.md(467 briefs)
Web
- https://www.beri.net/article/typesafe-jev-typed-decision-model-calibration-decomposition-shadow-eval — independent-study digest; source of the 0.107 off-distribution expected calibration error, the 62.6%-to-95.0% decomposition result, the Qwen3-4B LoRA community report, the shadow-eval methodology, and the vendor-versus-independent attribution split
- https://arxiv.org/pdf/2502.11830 — "Text Classification in the LLM Era"; ~10 labels per class for few-shot fine-tuning to beat zero-shot, and zero-shot weakening as class count grows
- https://arxiv.org/pdf/2406.08660 — "Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification"; ~200-label saturation point
- https://arxiv.org/pdf/2609.28940 — pentest decision-model preprint; cites expected calibration error 0.246 for Jev on TypeSafe's own evaluation set (vendor-set figure, not comparable across tasks)
- UCLA calibration audit of Jev 1.13.0 (Lingsen Meng), 3,244 evaluations over 2,372 texts, surfaced via search; the 0.071 phrasing-dependent calibration-error swing. Final per-model expected-calibration-error figures were not retrievable from the search excerpts and are unverified.