core: grow the retrieval eval until it can tell rerank models apart #920

Part of #918.

Context

The reranker is judged by brain eval --rerank jev (help text, packages/core/src/cli/commands/eval.ts:54-59) over packages/core/fixtures/corpus/evals/retrieval.jsonl: 24 queries on the example corpus. docs/extending/rerankers.md:143-145 says the jev rows scored within one or two queries of each other and that single-query differences are noise. A model comparison on that set can only find large gaps; Jev vs Clef vs Clef-flash may differ by less.

Already delivered, and not to be rebuilt here: every query carries a class (packages/core/fixtures/corpus/evals/retrieval.jsonl:2-25), aggregate emits one score row per class after the overall row (packages/core/src/lib/retrieval-eval.ts:322-338), and the human output prints them (printHuman, packages/core/src/cli/commands/eval.ts:440-464). What is missing is a tuning/held-out split, a set large enough to detect the differences #918 cares about, and a stated detection limit. The query schema is strict (querySchema, packages/core/src/lib/retrieval-eval.ts:76-87), so a split field is a set-format change.

Scope

• Measure the current set's run-to-run spread for jev-1.13.0 (keyless where possible from recorded answers; otherwise state exactly what a keyed run needs) and derive the smallest difference it can detect.
• Grow the set until it can detect a difference worth acting on, with query classes covering what reranking actually decides: lifecycle evidence (archived vs active, superseded), near-duplicate titles, questions vs exact titles, and candidates the first stage ranks badly.
• Represent the split in the set file: an optional per-query split field with values tuning or held-out, assigned by document, not by paraphrase (two queries whose expected documents overlap share a split). Accept it in querySchema and document it in docs/evaluating-search.md's field table.
• Report scores per split: brain eval adds per-split rows (overall and per class within each split) when the set declares splits, and behaves exactly as today when it does not.
• Keep the fixture corpus fictional (Odysseus world) and extend it where new queries need documents; update packages/core/fixtures/corpus/evals/expected-ranks.json and the retrieval goldens as a reviewed diff.

Out of scope

• Per-class reporting, which already exists.
• Changing ranking code, fusion constants (core: measure the #404 fusion constants with a keyed brain eval, before and after #468) or the default reranker.
• Running Clef or Clef-flash; that is the measurement task (core: measure Jev, Clef and Clef-flash as the brain sync judge #924 and its siblings).

Acceptance criteria

• Every query in retrieval.jsonl has a split; the assignment is by expected document, and a test asserts that no expected document appears in both splits.
• brain eval (human and --json) reports rows per split when splits are declared; a set without split produces byte-identical output to today, proven by an existing-set test.
• A short write-up next to the set (in docs/evaluating-search.md or beside the fixture) states its size, the measured spread and the minimum detectable difference in the eval's own metric, and says whether that spread came from recorded answers or a keyed run.
• Existing keyless eval tests still pass and cover the new classes and the split rows.

Verification

• bun run test packages/core/tests/eval-set.test.ts packages/core/tests/eval-cli.test.ts packages/core/tests/eval-corpus.test.ts keyless, then bun run test for the core package.
• brain eval --rerank heuristic over the grown set prints per-class and per-split rows; brain eval --json carries the split rows.
• Mutation: assign one shared-document query to the other split and watch the split-integrity test fail.

Contract impact

Additive. The query-set gains an optional split field and brain eval --json gains split rows (docs/integration-contract/cli.md, "brain eval --json"); update that component and the index in the same commit, prefix it CONTRACT:, label the PR contract, and ship a minor changeset. No existing field changes meaning, so schema_version stays.

2026-10-24: removed the already-delivered per-class reporting from the deliverables, specified how the split is represented and reported, replaced bare bun test with bun run test, and assessed the contract impact on main a6b945bd9.