Repository navigation
Attribute the SciFact BM25 gap to k1 and b - #9
Merged
Merged
Conversation
Document-level BM25 with the selected parameters is a diagnostic run after the frozen test numbers. It does not change a selection or a gate.
Quality metrics match the previous reports. index_ms is the first-stage index, including embedding initialization, and not the cross-encoder load.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Post-hoc attribution for the SciFact chunked-BM25 gap, plus a timing refresh of the committed rerank reports. Frozen selections, thresholds, and the existing headline test numbers are unchanged.
benchmarks/scifact/rerank-selected.jsonandbenchmarks/scifact/rerank-train.jsonwere regenerated with the current index clock. Cross-encoder weight loading is outsideindex_ms. Ranking metrics, including nDCG@10 and the rerank delta of -0.028, match the previous reports. Per-query latencies moved with the new wall clock, and the rendered p95 figures follow them.Test plan
ruff checkandruff format --checkmypypytestwith coverage (89.09% with the dense extra installed locally)docs/results.mddrift checks