ui-server: measure Jev, Clef and Clef-flash as the turn classifier #925

Blocked by #1606 and #922

Part of #918.

Context

Call site: packages/ui-server/src/app.ts:337-344 and the catalogue lines in packages/ui-sdk/src/classification/catalogue.ts:99. Once its eval can separate models and the runner can target Jev and Workers AI, this task measures jev-1.13.0, clef and clef-flash on it and recommends one.

This is the D42 answer-block classification pass (createTurnClassifier), which chooses presentation for assistant markdown; it is distinct from T1 intent routing in #848. Initial source discovery at main 1d1fcc72a7e06352e8b5ee020b3910864babc5db verifies the detector/request/catalogue ownership. planClassification (packages/ui-sdk/src/classification/request.ts:80-100) uses jev-latest, strips local thresholds before sending the request and batches detected candidates; record/pin the answered version for this experiment without changing production defaults. D45's contact route and exclusion of trend/number parsing also bind this comparison (docs/decisions/design-kit.md).

Use #922's real detector/planner plus applyClassification (packages/ui-sdk/src/classification/request.ts:103-121) to score actual anchored block outcomes. Report per-question accuracy, per-kind swap precision/coverage, wrong-swap count, should-stay-markdown cases and failures/fallbacks. Evaluate speculative companion questions only where their primary answer makes them relevant. Preserve bounded source text and deterministic transforms; no invented values or authorization decisions. Missing keys/timeouts/invalid responses/low confidence leave markdown visible under D42; preserve that baseline and the existing retry/deadline/breaker behavior when measuring end-to-end effects.

Prerequisites and recorded authorization

The maintainer granted the standing live-evaluation authorization on 2026-10-06, explicitly covering #923–#925 once their blockers close. Its $15 per-issue and $150 combined caps include retries, reruns, fallbacks and repetitions; the named models, secure environment credential access, subscription-first Claude billing and review by a different model family bind this work. Verify cumulative recorded spend before dispatch. No new authorization is needed within that scope, and no credential/account identifier belongs in public evidence. The independently reviewed golden/split receipt must identify its frozen fixture SHA before measurement; authored labels and scripted responses do not establish model quality. The eval and runner outputs remain prerequisites, not merely their closed states.

#1606 may complete keyless with its live wire check explicitly unrun. This task must therefore obtain the authorized wire receipt itself if missing: nonempty requests using the call site's real question/state shapes, transport parsing and usage/version recording for both Workers AI targets. Record envelope/schema divergences and any script-side normalization. A skipped target, bad_response or synthetic compatibility result is not a valid measured model. Package changes belong in a separate follow-up.

Scope

• Freeze the comparison protocol before held-out runs: fixture/corpus SHA, input generation, independent document/template split, quality gates, useful effect size, uncertainty method and stopping rule. Run the tuning split, then at least three repetitions per model on identical held-out inputs. Report sample and answered counts separately; repeated calls to one item are not independent additional fixtures. Estimate paired quality differences and uncertainty with the document/template grouping respected. A detection-limit estimate must state its assumptions; an observed difference below it is inconclusive, not evidence that models are equal.
• Measure current operating thresholds first where applicable. Any candidate threshold calibration uses only the tuning split and is frozen per model for the held-out report. It is an offline experiment, not a production threshold change; maintain the existing deterministic policy and safe fallback.
• Report quality by class/question/label with errors, abstentions and fallback coverage; include all failures/skips in denominators rather than silently dropping them. Record p50/p95 with sample counts and observed spread, and distinguish model-request time from end-to-end call-site time. Record every physical request/retry, billed input tokens, missing usage and cost per 1,000 attempted and successful calls using verified run-date prices. Unknown usage/cost remains unknown. For recommendations include fallback effects and any extra provider/embedding cost required by the selected lane.
• Write the result up in docs/ with a recommendation for the ui-server answer-block classifier: keep Jev, offer Clef(-flash) as a configured alternative, or switch the default. If evidence cannot support a choice, report "inconclusive, keep Jev". List any unresolved quality/latency/cost tradeoff for maintainer review in the proposed follow-up; this spike does not approve a default migration.

Out of scope

• Implementing the recommended change; file it as a follow-up and attach it to Epic: choose the decision model per use case — Jev, Clef or Clef-flash — on evals that can tell them apart #918.
• Other call sites.

Acceptance criteria

• Authorized provider/model/access/spend scope and independent-golden receipt are recorded before live work. Required outputs from both blocker issues and actual live compatibility are verified, rather than inferred from their closure.
• Report records code/fixture/corpus SHA, exact sanitized command/protocol, requested and answered model IDs (and any lack of an immutable provider version), dates/runtime, repetitions/sample counts, spread/uncertainty, failures/fallbacks, physical calls/tokens and linked run-date pricing. Detection limits state their assumptions; no unsupported equality conclusion.
• The current-policy baseline and any proposed operating points are reported separately. Thresholds used on held-out were frozen beforehand, or explicitly marked inapplicable for probability ordering; no production policy change occurs.
• Task-specific quality/error/coverage metrics in Context are reported per class/label/question. An insufficient or inconclusive run records its limitation and retains Jev.
• A recommendation is stated with its cost: what is lost in quality, latency or money against each alternative.
• Any recommended change is filed and attached to Epic: choose the decision model per use case — Jev, Clef or Clef-flash — on evals that can tell them apart #918.

Verification

An authorized runner reproduces the sanitized command against the recorded fixture/protocol/model versions and compares results using the reported uncertainty method; discrepancies remain explicit. Keyless replay validates nonempty scoring/report inputs and failure accounting without a network requirement in CI. No live result is claimed by this triage.

Contract impact

None. A change to defaults or config surface arrives with its follow-up.

2026-10-07 triage at main ef9a2bc01d12084fc952bc999dcefe41e9cdf6fc: folded the October 6 authorization into the specification. All declared runner/eval blockers remain open. The complete dependency-blocked brief may carry agent-ready alongside blocked; no live run or quality result is claimed.

Updated 2026-10-12: runner prerequisite now names #1606; retained the eval prerequisite, all unique comment rulings, authorization and actual-output proof requirements.

Updated 2026-10-12: the transferred compatibility requirements are consumed from #1606; measurements still require actual wire evidence or an explicit unavailable result.