From 336b3decdbba3725ea564098ff8b3d22e1de61fc Mon Sep 17 00:00:00 2001 From: rustfuture <121799572+rustfuture@users.noreply.github.com> Date: Wed, 30 Sep 2026 07:41:52 +0000 Subject: [PATCH 1/2] docs: align claims with committed evaluation artifacts - README: disclose the three v3 runs, their differing results and why run-03 is reported. - docs/llm-validation.md, evaluation/v2/README.md: run-03 is the current record; run-01 is historical. - Say precisely which failure mode the extractive contract removes and which remain. - evaluation/v3/README.md: use "previously exposed" for the 42 questions; document what changed between runs. - RELEASE_NOTES.md: use the committed benchmark and retrieval results; mark uncommitted earlier figures as historical. --- README.md | 9 ++++++++ RELEASE_NOTES.md | 16 +++++++++++--- docs/llm-validation.md | 18 ++++++++++------ evaluation/v2/README.md | 2 +- evaluation/v3/README.md | 47 +++++++++++++++++++++++++++++++++++------ 5 files changed, 75 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index f584f4d..1361e13 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,15 @@ questions (30 answerable, 12 unanswerable; 41 model calls, 1 pre-model refusal). Results yielded 12 false accepts (5 irrelevant, 2 partial, 5 on unanswerable questions) and 2 false rejects, with 21 expected sources fully covered and 7 correct refusals. +Three runs of these 42 questions are committed (`evaluation/v3/run-01`, `run-02`, `run-03`). +Run-03 is reported here because it is the latest run and the only one made after the unused +`src/citation.rs` module was removed (commit `79fc4c1`, clean tree); run-01 was made on a dirty +working tree and run-02 on commit `63c1eba`. The runs differ: false accepts / expected sources +covered / false rejects are 12 / 18 / 3 (run-01), 8 / 23 / 3 (run-02) and 12 / 21 / 2 (run-03), +so run-02 is the best of the three and run-03 is not the most favourable. Run-01 and run-02 record +identical source, question and corpus hashes and model digests, so that spread is run-to-run +variation rather than a recorded code change; treat these counts as one sample, not a stable rate. + The committed HTTP benchmark (`evaluation/v3/benchmark-local-http.json`) measures 200 sequential localhost requests on the fixed authored corpus at p50 0.857 ms / p95 0.920 ms (request overhead only, not generation latency). diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 7ba7760..7db1ddb 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -11,10 +11,20 @@ - Optional extractive-contract and single-fixture prompt-injection smoke scripts remain separate from normal CI because they require a local model; they check quotation integrity and one injected sentinel, not general injection immunity. - The stable GitHub Actions job runs the Rust suite and both deterministic retrieval evaluators; a separate job checks the Rust 1.85 MSRV. -Local sequential HTTP benchmark on 2026-09-14 (200 requests against the fixed authored corpus): p50 `0.448 ms`, p95 `0.566 ms`, max `0.675 ms`. These figures are localhost plumbing measurements, not production capacity claims. +Local sequential HTTP benchmark (200 requests against the fixed authored corpus), as committed in `evaluation/v3/benchmark-local-http.json`: p50 `0.857 ms`, p95 `0.920 ms`, max `2.110 ms`. These figures are localhost plumbing measurements, not production capacity claims. An earlier version of these notes gave p50 `0.448 ms` / p95 `0.566 ms` / max `0.675 ms` (2026-09-14, commit `2ecd148`) and, before that, `0.210 / 0.432 / 0.698 ms` (2026-09-06, commit `33d1a16`); no output file for either was committed, so they are not reproducible from the repository and are superseded by the committed artifact. -The product's deterministic retrieval run on the fixed 20-question corpus reported Recall@5 `1.00` and MRR `1.0000` for lexical, Recall@5 `1.00` and MRR `0.9375` for `HashEmbedding` semantic retrieval, and Recall@5 `1.00` and MRR `1.0000` for hybrid RRF. The authored corpus is a plumbing/regression fixture, not evidence of general retrieval quality. +Retrieval comparison, from the committed `evaluation/results_modes.json` (recorded 2026-09-16T20:01:04Z; corpus `evaluation/corpus`, 30 answerable questions of `evaluation/questions.json`; the 20-question column is the subset labelled `heldout` in that file, and all questions were previously exposed). Recall@5 is `1.00` for every mode on both sets. -The separate optional `evaluation/evaluate_hybrid.py` experiment uses local Ollama `nomic-embed-text`. Its current fixture result is lexical MRR `1.0000`, embedding MRR `0.9167`, and hybrid MRR `0.8667` (all Recall@5 `1.00`); this external comparison is not a product-quality claim. +| Mode | MRR, all 30 | MRR, 20-question subset | +|---|---:|---:| +| lexical | 1.0000 | 1.0000 | +| `HashEmbedding` semantic | 0.8428 | 0.7892 | +| `HashEmbedding` hybrid (RRF) | 0.9667 | 1.0000 | +| `nomic-embed-text` semantic | 0.9178 | 0.8767 | +| `nomic-embed-text` hybrid (RRF) | 0.9667 | 0.9750 | + +Lexical is the best or tied-best mode in both columns; neither hybrid mode beats it. The authored corpus is a plumbing/regression fixture, not evidence of general retrieval quality. + +Historical figures that no longer match a committed output: an earlier version of these notes reported `HashEmbedding` semantic MRR `0.9375` on a 20-question corpus (also recorded in `docs/implementation-report-2026-09-14.md`), and, from the optional `evaluation/evaluate_hybrid.py`, `nomic-embed-text` embedding MRR `0.9167` and hybrid MRR `0.8667`. They were measured on the 20-question `evaluation/questions.json` of commit `2ecd148` and earlier, before it was replaced by the 42-question set on 2026-09-16 (commit `277b1e1`); no output file for them is committed, and the current results are in the table above. `evaluation/evaluate_hybrid.py` embeds corpus lines through a local Ollama server and has no committed output. This is not a production service. Authentication, TLS, rate limiting, multi-tenant isolation, immutable commit-snapshot indexing, broad retrieval/answer evaluation, and AGY token/cost accounting remain outside this release candidate. diff --git a/docs/llm-validation.md b/docs/llm-validation.md index d076afb..590f60a 100644 --- a/docs/llm-validation.md +++ b/docs/llm-validation.md @@ -7,10 +7,15 @@ return only evidence IDs (`[E1]`, `[E2]`, …), at most three, or the exact toke verbatim with `path:line` references. Extra prose, malformed IDs, and mixed valid/invalid output are refused atomically. -This design removes the failure modes of the earlier free-form path, where an -answer was approved by word overlap against the cited span. Under extraction -there is no model-authored prose to contain an invented number, a flipped -negation, a reversed relation, or an added claim. +This design removes one failure mode of the earlier free-form path, where an +answer was approved by word overlap against the cited span: model-authored +prose. Under extraction there is no such prose to contain an invented number, +a flipped negation, a reversed relation, or an added claim. It does not remove +irrelevant selection: the model can still pick a real, exactly quoted excerpt +that does not answer the question, or accept a question that has no answer in +the repository. Those cases are counted as false accepts in the v3 runs (12 in +the reported run-03, see +[`evaluation/v3/run-03/summary.json`](../evaluation/v3/run-03/summary.json)). **What this does not establish.** An exact quotation is not evidence that the source is true, relevant, or answers the question. The model can select an @@ -18,8 +23,9 @@ irrelevant or even malicious repository file, and a selected document can assert something the code does not do. Quotation integrity and semantic entailment are different properties, and only the former is checked automatically. Old free-form validation notes and the 12/12 refusal claim are superseded; see -[`evaluation/v3/run-01/report.md`](../evaluation/v3/run-01/report.md) for the -current measured record. +[`evaluation/v3/run-03/report.md`](../evaluation/v3/run-03/report.md) for the +current measured record. Run-01 and run-02 are earlier runs of the same 42 +questions and remain committed as history; the README lists all three. ## Local model run diff --git a/evaluation/v2/README.md b/evaluation/v2/README.md index cddb98d..7929cf3 100644 --- a/evaluation/v2/README.md +++ b/evaluation/v2/README.md @@ -18,7 +18,7 @@ What was wrong with the v2 framing: application prints verbatim source text. Current evaluation: [`evaluation/v3`](../v3/README.md) and -[`evaluation/v3/run-01/report.md`](../v3/run-01/report.md). Do not pool v2 and v3 +[`evaluation/v3/run-03/report.md`](../v3/run-03/report.md). Do not pool v2 and v3 numbers. `report.md` and `summary.json` here are frozen v2 outputs and are unedited except diff --git a/evaluation/v3/README.md b/evaluation/v3/README.md index 3ba56e0..8a33b3c 100644 --- a/evaluation/v3/README.md +++ b/evaluation/v3/README.md @@ -12,14 +12,16 @@ This is the current evaluation for the answer path. It measures the | Path | Contents | |---|---| -| `questions.json` | 52 authored questions (10 dev, 42 held-out: 30 answerable, 12 unanswerable). Pinned `expected_sources` and `expected_answer` keys; no per-question code. | -| `evaluate.py` | Runs all 42 held-out questions against the local Ollama model and writes `raw.jsonl`, `summary.json`, `report.md`, `manifest.json`. `--render-only ` re-renders summary/report from `raw.jsonl` with **no model calls**. | +| `questions.json` | 52 authored questions (10 dev, 42 previously exposed and labelled `heldout` in the `split` field: 30 answerable, 12 unanswerable). Pinned `expected_sources` and `expected_answer` keys; no per-question code. | +| `evaluate.py` | Runs all 42 `heldout`-split questions against the local Ollama model and writes `raw.jsonl`, `summary.json`, `report.md`, `manifest.json`. `--render-only ` re-renders summary/report from `raw.jsonl` with **no model calls**. | | `test_evaluate.py` | Offline scoring-rule tests including the renderer. | -| `run-01/` | Frozen run: per-question raw records, derived summary and report, manifest with commit/dirty state, source/corpus/question SHA-256, and model digests. | +| `run-01/` | Historical run (dirty working tree at commit `522dd89`, which is not in the rewritten history): raw records, summary, report, manifest, and `manual-review.md`. 12 false accepts, 18 covered, 3 false rejects. | +| `run-02/` | Historical run on commit `63c1eba` (clean tree): 8 false accepts, 23 covered, 3 false rejects. | +| `run-03/` | The reported run, on commit `79fc4c1` (clean tree): 12 false accepts, 21 covered, 2 false rejects. Each run directory holds per-question raw records, derived summary and report, manifest with commit/dirty state, source/corpus/question SHA-256, and model digests. | ## Scoring rule (generic, data-driven) -For each held-out question the evaluator derives a verdict from the raw record: +For each `heldout`-split question the evaluator derives a verdict from the raw record: | Verdict | Meaning | |---|---| @@ -42,9 +44,40 @@ not about model or guard refusal behaviour). Source overlap is not entailment. `expected_source_covered` means the model selected the frozen expected span with an exact quotation; it does **not** prove the excerpt answers the question in natural language. Semantic correctness stays -a manual inspection (`run-01/manual-review.md`). The corpus is authored and the -questions were previously exposed, so this is a regression record, not -unseen-generalization evidence. No thresholds were tuned on the reported run. +a manual inspection; the only manual review committed is for the historical +run-01 (`run-01/manual-review.md`), covers six questions, and was not repeated +for run-03. The corpus is authored and the questions were previously exposed, so +this is a regression record, not unseen-generalization evidence. + +The `report.md` files are generated by `evaluate.py` and label the 42 questions +"held-out" after their `split` value; they were previously exposed, as the +manifests' `limitation` field says. + +## Three runs + +Run-01, run-02 and run-03 use the same questions (`questions_sha256`), corpus +and model digests (see each `manifest.json`). What differed: + +- run-01 and run-02 record identical source hashes (`src/citation.rs`, + `extractive.rs`, `lib.rs`, `llm.rs`, `main.rs`). Run-01 was taken on a dirty + working tree; run-02 was taken on clean commit `63c1eba` for pristine + provenance. Their results differ (12 vs 8 false accepts) with no recorded + code difference. +- between run-02 (`63c1eba`) and run-03 (`79fc4c1`) the source changed only by + removing the unused `src/citation.rs` module, moving the `ClaimAssessment` and + `AnswerAssessment` types into `src/extractive.rs`, and updating their paths in + `src/lib.rs` and `src/main.rs`; the model prompt (`INSTRUCTION`) and + `src/llm.rs` are unchanged, and `evaluation/v3/evaluate.py` gained a `sys` + import. + +The selection path has no score thresholds (none appear in `src/`), and the committed history shows no change +to the scoring rule, questions, corpus or model prompt between runs +(`evaluate.py` gained only a refusal to run on a dirty tree in `63c1eba` and a +`sys` import in `79fc4c1`), so "no thresholds were tuned" holds for each run. +It does not mean a single run was made: three were, and the spread across +them (false accepts 8 to 12, covered 18 to 23) is the run-to-run variation to +read the reported run against. Run-03 is the reported run because it is the +latest and post-dates the module removal, not because it is the best; run-02 is. ## Reproduce From d2c92453971471a87d4139f0be4321a6505f10a1 Mon Sep 17 00:00:00 2001 From: rustfuture <121799572+rustfuture@users.noreply.github.com> Date: Wed, 30 Sep 2026 07:46:28 +0000 Subject: [PATCH 2/2] docs: reword report-rendering note --- evaluation/v3/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evaluation/v3/README.md b/evaluation/v3/README.md index 8a33b3c..2c08a0c 100644 --- a/evaluation/v3/README.md +++ b/evaluation/v3/README.md @@ -49,7 +49,7 @@ run-01 (`run-01/manual-review.md`), covers six questions, and was not repeated for run-03. The corpus is authored and the questions were previously exposed, so this is a regression record, not unseen-generalization evidence. -The `report.md` files are generated by `evaluate.py` and label the 42 questions +The `report.md` files are rendered from `raw.jsonl` by `evaluate.py` and label the 42 questions "held-out" after their `split` value; they were previously exposed, as the manifests' `limitation` field says.