Skip to content

Attribute the SciFact BM25 gap to k1 and b - #9

Merged
Shuvam1024 merged 2 commits into
mainfrom
shuvam1024/k1b-ablation
Oct 7, 2026
Merged

Shuvam1024 merged 2 commits into
mainfrom
shuvam1024/k1b-ablation

Conversation

@Shuvam1024

@Shuvam1024 Shuvam1024 commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Summary

Post-hoc attribution for the SciFact chunked-BM25 gap, plus a timing refresh of the committed rerank reports. Frozen selections, thresholds, and the existing headline test numbers are unchanged.

  • Document-level BM25 with the selected parameters (k1 1.5, b 0.75, stem) is scored on the SciFact test split and on NFCorpus. These runs were done after the frozen test numbers were reported. They are not a train selection and they do not replace the document-level baseline or the selected chunked index.
  • On SciFact, that ablation's nDCG@10 is 0.6910, against 0.6741 for document-level BM25 at k1 0.9, b 0.4 (mean delta +0.017, 95% CI [+0.005, +0.028], sign-flip p 0.0045) and 0.6907 for the selected chunked index. The chunked index differs on 5 of 300 queries (mean delta -0.0003, 95% CI [-0.0007, -0.00004], p 0.0621). The +0.017 gap is the parameter change. Chunking adds about nothing on this split.
  • NFCorpus gets the same ablation. The parameter-change interval includes zero. Chunking versus the matched parameters also leaves the nDCG interval covering zero (12 of 323 queries differ).
  • benchmarks/scifact/rerank-selected.json and benchmarks/scifact/rerank-train.json were regenerated with the current index clock. Cross-encoder weight loading is outside index_ms. Ranking metrics, including nDCG@10 and the rerank delta of -0.028, match the previous reports. Per-query latencies moved with the new wall clock, and the rendered p95 figures follow them.

Test plan

  • ruff check and ruff format --check
  • mypy
  • pytest with coverage (89.09% with the dense extra installed locally)
  • README and docs/results.md drift checks
  • CI jobs on this pull request

Document-level BM25 with the selected parameters is a diagnostic run
after the frozen test numbers. It does not change a selection or a gate.
Quality metrics match the previous reports. index_ms is the first-stage
index, including embedding initialization, and not the cross-encoder load.
@Shuvam1024
Shuvam1024 marked this pull request as ready for review October 7, 2026 20:45
@Shuvam1024
Shuvam1024 merged commit 972455e into main Oct 7, 2026
5 checks passed
@Shuvam1024
Shuvam1024 deleted the shuvam1024/k1b-ablation branch October 7, 2026 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant