docs: align claims with committed evaluation artifacts

README.md

@@ -87,6 +87,15 @@ questions (30 answerable, 12 unanswerable; 41 model calls, 1 pre-model refusal).
 Results yielded 12 false accepts (5 irrelevant, 2 partial, 5 on unanswerable questions)
 and 2 false rejects, with 21 expected sources fully covered and 7 correct refusals.
 
+Three runs of these 42 questions are committed (`evaluation/v3/run-01`, `run-02`, `run-03`).
+Run-03 is reported here because it is the latest run and the only one made after the unused
+`src/citation.rs` module was removed (commit `79fc4c1`, clean tree); run-01 was made on a dirty
+working tree and run-02 on commit `63c1eba`. The runs differ: false accepts / expected sources
+covered / false rejects are 12 / 18 / 3 (run-01), 8 / 23 / 3 (run-02) and 12 / 21 / 2 (run-03),
+so run-02 is the best of the three and run-03 is not the most favourable. Run-01 and run-02 record
+identical source, question and corpus hashes and model digests, so that spread is run-to-run
+variation rather than a recorded code change; treat these counts as one sample, not a stable rate.
+
 The committed HTTP benchmark (`evaluation/v3/benchmark-local-http.json`) measures 200
 sequential localhost requests on the fixed authored corpus at p50 0.857 ms / p95 0.920 ms
 (request overhead only, not generation latency).

RELEASE_NOTES.md

@@ -11,10 +11,20 @@
 - Optional extractive-contract and single-fixture prompt-injection smoke scripts remain separate from normal CI because they require a local model; they check quotation integrity and one injected sentinel, not general injection immunity.
 - The stable GitHub Actions job runs the Rust suite and both deterministic retrieval evaluators; a separate job checks the Rust 1.85 MSRV.
 
-Local sequential HTTP benchmark on 2026-09-14 (200 requests against the fixed authored corpus): p50 `0.448 ms`, p95 `0.566 ms`, max `0.675 ms`. These figures are localhost plumbing measurements, not production capacity claims.
+Local sequential HTTP benchmark (200 requests against the fixed authored corpus), as committed in `evaluation/v3/benchmark-local-http.json`: p50 `0.857 ms`, p95 `0.920 ms`, max `2.110 ms`. These figures are localhost plumbing measurements, not production capacity claims. An earlier version of these notes gave p50 `0.448 ms` / p95 `0.566 ms` / max `0.675 ms` (2026-09-14, commit `2ecd148`) and, before that, `0.210 / 0.432 / 0.698 ms` (2026-09-06, commit `33d1a16`); no output file for either was committed, so they are not reproducible from the repository and are superseded by the committed artifact.
 
-The product's deterministic retrieval run on the fixed 20-question corpus reported Recall@5 `1.00` and MRR `1.0000` for lexical, Recall@5 `1.00` and MRR `0.9375` for `HashEmbedding` semantic retrieval, and Recall@5 `1.00` and MRR `1.0000` for hybrid RRF. The authored corpus is a plumbing/regression fixture, not evidence of general retrieval quality.
+Retrieval comparison, from the committed `evaluation/results_modes.json` (recorded 2026-09-16T20:01:04Z; corpus `evaluation/corpus`, 30 answerable questions of `evaluation/questions.json`; the 20-question column is the subset labelled `heldout` in that file, and all questions were previously exposed). Recall@5 is `1.00` for every mode on both sets.
 
-The separate optional `evaluation/evaluate_hybrid.py` experiment uses local Ollama `nomic-embed-text`. Its current fixture result is lexical MRR `1.0000`, embedding MRR `0.9167`, and hybrid MRR `0.8667` (all Recall@5 `1.00`); this external comparison is not a product-quality claim.
+| Mode | MRR, all 30 | MRR, 20-question subset |
+|---|---:|---:|
+| lexical | 1.0000 | 1.0000 |
+| `HashEmbedding` semantic | 0.8428 | 0.7892 |
+| `HashEmbedding` hybrid (RRF) | 0.9667 | 1.0000 |
+| `nomic-embed-text` semantic | 0.9178 | 0.8767 |
+| `nomic-embed-text` hybrid (RRF) | 0.9667 | 0.9750 |
+
+Lexical is the best or tied-best mode in both columns; neither hybrid mode beats it. The authored corpus is a plumbing/regression fixture, not evidence of general retrieval quality.
+
+Historical figures that no longer match a committed output: an earlier version of these notes reported `HashEmbedding` semantic MRR `0.9375` on a 20-question corpus (also recorded in `docs/implementation-report-2026-09-14.md`), and, from the optional `evaluation/evaluate_hybrid.py`, `nomic-embed-text` embedding MRR `0.9167` and hybrid MRR `0.8667`. They were measured on the 20-question `evaluation/questions.json` of commit `2ecd148` and earlier, before it was replaced by the 42-question set on 2026-09-16 (commit `277b1e1`); no output file for them is committed, and the current results are in the table above. `evaluation/evaluate_hybrid.py` embeds corpus lines through a local Ollama server and has no committed output.
 
 This is not a production service. Authentication, TLS, rate limiting, multi-tenant isolation, immutable commit-snapshot indexing, broad retrieval/answer evaluation, and AGY token/cost accounting remain outside this release candidate.

docs/llm-validation.md

@@ -7,19 +7,25 @@ return only evidence IDs (`[E1]`, `[E2]`, …), at most three, or the exact toke
 verbatim with `path:line` references. Extra prose, malformed IDs, and mixed
 valid/invalid output are refused atomically.
 
-This design removes the failure modes of the earlier free-form path, where an
-answer was approved by word overlap against the cited span. Under extraction
-there is no model-authored prose to contain an invented number, a flipped
-negation, a reversed relation, or an added claim.
+This design removes one failure mode of the earlier free-form path, where an
+answer was approved by word overlap against the cited span: model-authored
+prose. Under extraction there is no such prose to contain an invented number,
+a flipped negation, a reversed relation, or an added claim. It does not remove
+irrelevant selection: the model can still pick a real, exactly quoted excerpt
+that does not answer the question, or accept a question that has no answer in
+the repository. Those cases are counted as false accepts in the v3 runs (12 in
+the reported run-03, see
+[`evaluation/v3/run-03/summary.json`](../evaluation/v3/run-03/summary.json)).
 
 **What this does not establish.** An exact quotation is not evidence that the
 source is true, relevant, or answers the question. The model can select an
 irrelevant or even malicious repository file, and a selected document can assert
 something the code does not do. Quotation integrity and semantic entailment are
 different properties, and only the former is checked automatically. Old
 free-form validation notes and the 12/12 refusal claim are superseded; see
-[`evaluation/v3/run-01/report.md`](../evaluation/v3/run-01/report.md) for the
-current measured record.
+[`evaluation/v3/run-03/report.md`](../evaluation/v3/run-03/report.md) for the
+current measured record. Run-01 and run-02 are earlier runs of the same 42
+questions and remain committed as history; the README lists all three.
 
 ## Local model run
 

evaluation/v2/README.md

@@ -18,7 +18,7 @@ What was wrong with the v2 framing:
   application prints verbatim source text.
 
 Current evaluation: [`evaluation/v3`](../v3/README.md) and
-[`evaluation/v3/run-01/report.md`](../v3/run-01/report.md). Do not pool v2 and v3
+[`evaluation/v3/run-03/report.md`](../v3/run-03/report.md). Do not pool v2 and v3
 numbers.
 
 `report.md` and `summary.json` here are frozen v2 outputs and are unedited except

evaluation/v3/README.md

@@ -12,14 +12,16 @@ This is the current evaluation for the answer path. It measures the
 
 | Path | Contents |
 |---|---|
-| `questions.json` | 52 authored questions (10 dev, 42 held-out: 30 answerable, 12 unanswerable). Pinned `expected_sources` and `expected_answer` keys; no per-question code. |
-| `evaluate.py` | Runs all 42 held-out questions against the local Ollama model and writes `raw.jsonl`, `summary.json`, `report.md`, `manifest.json`. `--render-only <run-dir>` re-renders summary/report from `raw.jsonl` with **no model calls**. |
+| `questions.json` | 52 authored questions (10 dev, 42 previously exposed and labelled `heldout` in the `split` field: 30 answerable, 12 unanswerable). Pinned `expected_sources` and `expected_answer` keys; no per-question code. |
+| `evaluate.py` | Runs all 42 `heldout`-split questions against the local Ollama model and writes `raw.jsonl`, `summary.json`, `report.md`, `manifest.json`. `--render-only <run-dir>` re-renders summary/report from `raw.jsonl` with **no model calls**. |
 | `test_evaluate.py` | Offline scoring-rule tests including the renderer. |
-| `run-01/` | Frozen run: per-question raw records, derived summary and report, manifest with commit/dirty state, source/corpus/question SHA-256, and model digests. |
+| `run-01/` | Historical run (dirty working tree at commit `522dd89`, which is not in the rewritten history): raw records, summary, report, manifest, and `manual-review.md`. 12 false accepts, 18 covered, 3 false rejects. |
+| `run-02/` | Historical run on commit `63c1eba` (clean tree): 8 false accepts, 23 covered, 3 false rejects. |
+| `run-03/` | The reported run, on commit `79fc4c1` (clean tree): 12 false accepts, 21 covered, 2 false rejects. Each run directory holds per-question raw records, derived summary and report, manifest with commit/dirty state, source/corpus/question SHA-256, and model digests. |
 
 ## Scoring rule (generic, data-driven)
 
-For each held-out question the evaluator derives a verdict from the raw record:
+For each `heldout`-split question the evaluator derives a verdict from the raw record:
 
 | Verdict | Meaning |
 |---|---|
@@ -42,9 +44,40 @@ not about model or guard refusal behaviour).
 Source overlap is not entailment. `expected_source_covered` means the model
 selected the frozen expected span with an exact quotation; it does **not** prove
 the excerpt answers the question in natural language. Semantic correctness stays
-a manual inspection (`run-01/manual-review.md`). The corpus is authored and the
-questions were previously exposed, so this is a regression record, not
-unseen-generalization evidence. No thresholds were tuned on the reported run.
+a manual inspection; the only manual review committed is for the historical
+run-01 (`run-01/manual-review.md`), covers six questions, and was not repeated
+for run-03. The corpus is authored and the questions were previously exposed, so
+this is a regression record, not unseen-generalization evidence.
+
+The `report.md` files are rendered from `raw.jsonl` by `evaluate.py` and label the 42 questions
+"held-out" after their `split` value; they were previously exposed, as the
+manifests' `limitation` field says.
+
+## Three runs
+
+Run-01, run-02 and run-03 use the same questions (`questions_sha256`), corpus
+and model digests (see each `manifest.json`). What differed:
+
+- run-01 and run-02 record identical source hashes (`src/citation.rs`,
+  `extractive.rs`, `lib.rs`, `llm.rs`, `main.rs`). Run-01 was taken on a dirty
+  working tree; run-02 was taken on clean commit `63c1eba` for pristine
+  provenance. Their results differ (12 vs 8 false accepts) with no recorded
+  code difference.
+- between run-02 (`63c1eba`) and run-03 (`79fc4c1`) the source changed only by
+  removing the unused `src/citation.rs` module, moving the `ClaimAssessment` and
+  `AnswerAssessment` types into `src/extractive.rs`, and updating their paths in
+  `src/lib.rs` and `src/main.rs`; the model prompt (`INSTRUCTION`) and
+  `src/llm.rs` are unchanged, and `evaluation/v3/evaluate.py` gained a `sys`
+  import.
+
+The selection path has no score thresholds (none appear in `src/`), and the committed history shows no change
+to the scoring rule, questions, corpus or model prompt between runs
+(`evaluate.py` gained only a refusal to run on a dirty tree in `63c1eba` and a
+`sys` import in `79fc4c1`), so "no thresholds were tuned" holds for each run.
+It does not mean a single run was made: three were, and the spread across
+them (false accepts 8 to 12, covered 18 to 23) is the run-to-run variation to
+read the reported run against. Run-03 is the reported run because it is the
+latest and post-dates the module removal, not because it is the best; run-02 is.
 
 ## Reproduce