Epic: choose the decision model per use case — Jev, Clef or Clef-flash — on evals that can tell them apart #918

Outcome

Each place brain-kit asks a decision model runs on the model that measured best for that use case — Jev, Clef or Clef-flash — chosen on an eval that is large and specific enough to tell the models apart, with quality, latency and cost recorded side by side. A use case where the eval cannot separate them says so and keeps today's default.

Why now

Cloudflare released Clef and Clef-flash on 2026-10-01 with System One typed questions. The Clef model page and Clef-flash page list 27B/9B models, 65,536-token context windows and input prices of $0.24/$0.09 per million tokens. TypeSafe's model reference lists Jev at $0.042 per million input tokens, a 64k total request budget and a 32k state-plus-longest-question budget, with text-only input. Verified 2026-10-03; recheck prices and availability at run time. Published vendor latency/quality benchmarks are motivation, not measurements of our tasks.

Current keyless source discovery at main 1d1fcc72a7e06352e8b5ee020b3910864babc5db confirms 24 retrieval queries through the real parseEvalSet loader, and 40 file plus 58 pair sync fixtures. The historical one/two-query spread in docs/extending/rerankers.md:116-130 describes separate 27/83-query measurements on a different corpus; it does not establish the public 24-query set's detection limit. The answer-block classifier needs the labelled eval in #922.

Request-schema compatibility does not establish unchanged transport compatibility. The Workers AI REST guide documents a result/success envelope. A keyless injected-fetch probe with two nonempty Choice/Noul questions at the SHA above found both existing clients return answered for flat {model, answers, usage} and bad_response for the same payload inside that envelope (createJevClient, packages/core/src/lib/jev.ts:248-354; createJevClient, packages/ui-server/src/classification/jev-client.ts:94-209). This proves parser behavior, not Clef's live response. #1606 owns shared-runner compatibility discovery; each measurement must obtain authorized live wire receipts or record why comparison remains unavailable. No package transport/default migration is approved here.

Shared context

The three call sites (current construction anchors verified at main af0547c2d4d197ed3766f2817cf56eb5f0486ab4; the discovery measurements above retain their historical SHA):

1. Search reranker — packages/core/src/providers/rerankers/jev.ts:226; config already takes model, apiKeyEnv, baseUrl (:73-82); pinned model packages/core/src/lib/llm-defaults.ts:28; eval brain eval --rerank jev over packages/core/fixtures/corpus/evals/retrieval.jsonl.
2. brain sync judge — packages/core/src/lib/sync/judge.ts; options include an injected client but no model/endpoint setting (SyncJudgeOptions, packages/core/src/lib/sync/judge.ts:565-575); the client has endpoint (packages/core/src/lib/jev.ts:251); fixtures packages/core/fixtures/sync-judge/{files,pairs}.jsonl; thresholds docs/decisions/deterministic-sync.md:74-75; measured by scripts/measure-sync-judge.ts.
3. ui-server answer-block classifier (createTurnClassifier, D42; distinct from T1 intent triage in ui-server: evaluate JEV as the bounded T1 triage classifier #848) — packages/ui-server/src/app.ts:531-539 constructs the client with apiKey/log and no model/endpoint configuration; client endpoint option at packages/ui-server/src/classification/jev-client.ts:70,102; questions and confidence lines in packages/ui-sdk/src/classification/catalogue.ts:99,573. The reranker uses the pinned model above. On main today the sync judge requests the shared client's jev-latest alias (packages/core/src/lib/jev.ts:34, used at packages/core/src/lib/sync/judge.ts:247,257) until core: the sync judge requests the moving jev-latest alias while the reranker pins jev-1.13.0, through two Jev transports and two JEV_MODEL constants #2504 lands its approved jev-1.13.0 pin; answer-block planning uses jev-latest (packages/ui-sdk/src/classification/request.ts:29). Measure a recorded version rather than allowing alias drift.

This epic follows the discovery and evaluation protocol in #838 (tuning/held-out splits, thresholds frozen before held-out runs, recorded fixture SHA, model version, repetitions and spread, verified pricing at run time). Candidate Jev uses still being evaluated under #838 (#847, #848 and the others) are not duplicated here: once one is adopted, its eval joins the same model matrix through the shared harness below.

Constraints and non-goals

• No default provider or model changes until the measurement task for that call site recommends one; the switch is its own follow-up. One approved exception (maintainer, 2026-10-22, on core: the sync judge requests the moving jev-latest alias while the reranker pins jev-1.13.0, through two Jev transports and two JEV_MODEL constants #2504): the brain sync judge is pinned to jev-1.13.0 before core: measure Jev, Clef and Clef-flash as the brain sync judge #924 completes, because its thresholds were measured on that version (docs/decisions/deterministic-sync.md:145). The pin is a requested-model correction, not a quality claim; it changes no other call site and does not replace core: measure Jev, Clef and Clef-flash as the brain sync judge #924's comparison or later recommendation. core: the sync judge requests the moving jev-latest alias while the reranker pins jev-1.13.0, through two Jev transports and two JEV_MODEL constants #2504 implements it.
• No new generic "decision provider" seam; scoped switch proposals must follow ROADMAP.md and the not-pluggable list in docs/extending/README.md. Provider absence stays opt-in and keyless CI stays keyless.
• Fixtures are public and fictional under AGENTS.md and docs/decisions/example-corpus.md: Odysseus is the sole world across core, CLI, UI and measurements. No real-brain content or account/deployment identifiers in public artifacts.
• Self-hosting the open weights, image input and the score type are out of scope; a discovered need requires a separate scoped proposal.
• The maintainer granted the standing live-evaluation authorization on 2026-10-06, explicitly covering core: measure Jev, Clef and Clef-flash as the search reranker #923–ui-server: measure Jev, Clef and Clef-flash as the turn classifier #925 once their blockers close. Its $15 per-issue and $150 combined caps include retries, reruns, fallbacks and repetitions; the named models, secure environment credential access, subscription-first Claude billing and review by a different model family bind this work. Verify cumulative recorded spend before dispatch. No new authorization is needed within that scope, and no credential/account identifier belongs in public evidence. The independently reviewed golden/split receipt must identify its frozen fixture SHA before measurement; authored labels and scripted responses do not establish model quality. The eval and runner outputs remain prerequisites, not merely their closed states.

Breakdown

The retained native children own evals and comparisons; shared runner #1606 is owned by #1603 and consumed here. #919 is consolidated into it. The epic is coordination and is never coded directly:

1. Consume core: share one model/provider/effort matrix runner across eval harnesses #1606 — the shared, keyless-testable model/provider/effort matrix runner and transport discovery; use its compatible immutable results, per-target records, wire receipts and explicit incomplete-cell status.
2. core: grow the retrieval eval until it can tell rerank models apart #920 — retrieval eval; core: add a held-out set to the sync judge eval #921 — held-out sync judgments; ui-sdk: add a labelled eval for the turn classification catalogue #922 — answer-block catalogue eval.
3. core: measure Jev, Clef and Clef-flash as the search reranker #923 — reranker comparison (dependent on core: share one model/provider/effort matrix runner across eval harnesses #1606/core: grow the retrieval eval until it can tell rerank models apart #920); core: measure Jev, Clef and Clef-flash as the brain sync judge #924 — sync judge comparison (dependent on core: share one model/provider/effort matrix runner across eval harnesses #1606/core: add a held-out set to the sync judge eval #921); ui-server: measure Jev, Clef and Clef-flash as the turn classifier #925 — answer-block comparison (dependent on core: share one model/provider/effort matrix runner across eval harnesses #1606/ui-sdk: add a labelled eval for the turn classification catalogue #922).

Prepare fixtures/harnesses independently. Measurement reports use independently reviewed goldens, fixed inputs and a protocol frozen before held-out evaluation, with per-case quality gates, useful effect sizes and uncertainty methods stated. Repetitions measure call variability; they do not create independent new documents. Differences below an estimated detection limit are inconclusive, not proof of equivalence. Keep Jev if the evidence cannot support a switch; any quality/latency/cost tradeoff requiring a ruling remains explicit in the proposed follow-up.

Definition of done

Each call site has an independently reviewed tuning/held-out eval and a reproducible measurement report stating sample counts, run-to-run spread, the method/assumptions for its detectable effect estimate, quality, p50/p95 latency and cost per 1,000 calls. The report recommends a model or records "inconclusive, keep Jev", with limitations and any unresolved tradeoff. Unavailable live evidence is not a completed comparison. Every recommended switch is filed and attached here; production migration follows its scoped approval and implementation. No new CLI/MCP/HTTP/wire/config contract is created by this coordination epic.

2026-10-07 triage at main ef9a2bc01d12084fc952bc999dcefe41e9cdf6fc: reconciled the later paid-evaluation authorization and sole-Odysseus ruling; refreshed classifier construction. Epics remain without agent-ready, and child runner/eval prerequisites remain active.

2026-10-07 source reconciliation at main 86ca7f2408993113f3cba4287b7a01846a8b26ee: refreshed the named current-source anchors. Historical measurement/implementation receipts retain their original provenance; no new design or runtime completion is inferred.

2026-10-10 triage at main a1e007c473cde021a472917a66fe56ec531a3cd8: refreshed named current-source pointers after the coordinator/control/demo deliveries. Original immutable failure receipts, unresolved prerequisites and approved behavior remain; source inspection supplies no new runtime or CI completion proof.

2026-10-10 source correction at main f1aa4c2c7a2d2fa87f669e6a8bdd9fdc71e0f41b: refreshed moved source anchors after the durable inbox bridge addition. Existing scope, prerequisites and immutable runtime receipts remain unchanged; source inspection supplies no new runtime proof.

2026-10-11 source correction at main 8c4b2411df64a8cf8c2dfa05d7757c2509c74f13: refreshed the classifier construction anchor after gated Queue dispatch wiring. Existing measurement scope, priorities and immutable receipts remain; no new model/default or measurement result is selected.

Updated 2026-10-12: the model-selection epic consumes #1606 outputs; #919 consolidation transfers runner ownership to #1603 without changing measurement authorization or completeness requirements.

2026-10-13 source reconciliation at main 857ab01d1f9cb591f928755f5be516ab7f73338e: refreshed named current-source anchors after governed-flow integration. Historical receipts, scope and remaining prerequisites are unchanged; source inspection supplies no new runtime proof.

2026-10-14 source reconciliation at main 4af7b2f20a43d23260c2ee5e3bac3896aee971bd: verified and refreshed the active named source anchors after #1793. Historical receipts remain historical; native storage delivery does not complete the other child outcomes, enable autonomous/scheduled dispatch or establish new runtime proof in this triage.

2026-10-16 triage at main ed6b0c5f09ada1b6f899bf245f3f1b176352bf77: Refreshed current classifier endpoint and model-constant declaration pointers. The historical parser probe keeps its original source identity. Existing evaluation inputs, authorization limits, native tasks and no-epic-readiness rule are unchanged; no model request or comparison was run.

2026-10-16 triage at main 4ba792473152c89eb74c286fc914e54ca1dae8fc: reverified the answer-block classifier construction and moved its pointer. Evaluation demand, transport limitations and approved model-selection protocol remain unchanged; no live measurement or model migration was performed.

2026-10-20 triage at main af0547c2d4d197ed3766f2817cf56eb5f0486ab4: refreshed the active answer-block classifier construction and endpoint-option declaration pointers. Historical parser probes, model/pricing observations and execution receipts retain their original identities; no current vendor capability, paid comparison or model migration was verified or executed. Native evaluation/runner prerequisites and no-epic-readiness status remain.

2026-10-22 source audit at a310317: refreshed the createTurnClassifier/client-construction range to app.ts:514-523 after per-app state moved it. Frozen historical parser/measurement receipts retain their original identities. The selected evaluation protocol, authorization caps, runner/eval prerequisites and no-epic-readiness rule remain; no paid comparison or model migration is executed.

2026-10-22 source navigation at main 2ba99c607dec41f95be5901fc8f309611806ceb5: verified the actual answer-block classifier construction and refreshed its moved active anchors after #2464. Historical source-bound receipts, scope, decisions, labels and remaining prerequisites are unchanged; no new execution or activation is claimed.

2026-10-22 source navigation at main aeaaf7794b6be41fde4e01bef78ce3f1a5828ac6: verified and refreshed the moved active app-construction citation(s) after #2397. Historical source-bound receipts, scope, decisions and prerequisites are unchanged; no new runtime, activation or epic-completion evidence is claimed.

2026-10-23 source navigation at main f3c6fdc9f9c6321fda5a384cf8cfd8bf2e4846da: verified and refreshed moved active app-construction anchors after reference-source composition. Historical source-bound receipts, scope, decisions and prerequisites remain unchanged; no new runtime, autonomous activation or epic completion is claimed.

2026-10-24: folded the maintainer's 2026-10-22 ruling on #2504 into Constraints as the single approved exception to the measurement-first rule (sync judge pinned to jev-1.13.0 before #924 completes), and stated what the sync judge requests on main until #2504 lands.