Repository navigation
Name the quality-gate policies and show a 0.07 drop fail - #3
Merged
Merged
Conversation
The whitespace tokenizer scores SciFact nDCG@10 at 0.643. An optional stem tokenizer drops a vendored English stopword list and applies the Snowball English stemmer; on the unchanged test split that reaches 0.683. The default stays whitespace so existing configs and the support fixture keep their scores. ragbench dataset --split writes a non-default qrels split beside benchmark.json instead of replacing it. SciFact train has 809 queries, so hybrid dense_weight can be chosen there and reported on the 300-query test set. pytrec_eval now checks Recall, Precision, MAP, MRR, and binary nDCG on a fixed fixture. Graded nDCG stays on exponential gain, which matches pytrec_eval only when every grade is 1. Co-authored-by: Shuvam Mukherjee <[email protected]>
Chunk size, stemming, and hybrid weight are chosen on the 809-query train split and committed before the test split is scored again. Document-level BM25 is the named comparison baseline, not a Pyserini run. NFCorpus is registered only for a later confirmation. Earlier test-split runs stay labeled as exploratory, and metric checks now include a pytrec_eval fixture. Co-authored-by: Shuvam Mukherjee <[email protected]>
The train-selected chunked BM25, document-level BM25, and hybrid systems were scored once on the SciFact test split and repeated on NFCorpus. Paired intervals are included. The chunked BM25 edge over document-level BM25 is small and does not repeat across metrics on NFCorpus. The hybrid interval stays above zero on both corpora. Co-authored-by: Shuvam Mukherjee <[email protected]>
point_drop stays the CI rule. proven_regression fails only when the upper confidence bound shows a drop past the margin. non_inferior fails unless the lower bound stays inside that margin.
The negative-control script and Actions job expect compare to exit 2 and write the metric table to the job summary. The workflow stays green when that rejection happens.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #2. This branch starts at
3053ee5. The gate work is3279686and258ee23. Until #2 merges, the diff againstmainalso contains that SciFact work.What changed
CI still uses point-drop thresholds.
configs/thresholds.yamlandconfigs/scifact-thresholds.yamlkeepgate_on_ci: false. A drop larger thanmax_dropfails. The support fixture allows 0.02 on MRR, Recall@1, and Recall@3.gate_modenames two different interval policies. They are not the CI rule.point_dropmax_drop. This is CI.proven_regression(candidate - baseline)is below-max_drop.gate_on_ci: trueis the older spelling of this mode.non_inferior(candidate - baseline)is at least-max_drop.scripts/demonstrate_gate_failure.pylowers the committed support-fixture baseline by 0.07 on those three metrics.ragbench compareagainstconfigs/thresholds.yamlexits 2. The script exits 0 when that rejection happens, and writes the metric table toGITHUB_STEP_SUMMARYwhen that variable is set.The Actions job
Negative control gateruns that script. The happy-path support-fixture and SciFact compares are unchanged and still fail the workflow on exit 2. The negative-control job stays green only because the gate rejected the degraded report.How to verify
The script exits 0. The compare command exits 2 and prints
FAILfor MRR, Recall@1, and Recall@3.In Actions, open the
Negative control gatejob. The step summary has the metric table. The Result column isFAILunderpoint_drop.Behavior worth knowing
On this 27-question fixture the 0.07 mean drop is carried by a few questions. The MRR interval is about [-0.177, 0.000].
point_dropfails.proven_regressionwould pass, because the upper bound is not below -0.02.non_inferiorwould fail. That is why CI stays on the point estimate: the upper-bound rule does not reject this regression. A 300-query set narrows the same kind of interval. Details are indocs/regression.md.