Add the threshold-shift drift gate and a paired head-to-head comparison - #6
Merged
Merged
Conversation
--fail-on threshold-shift=X fails when the cost-optimal line moves by more than X in either direction between the compared versions. Without a cost matrix there is no line on either side, so the check is refused with exit 1 instead of passing. The CI block now names each failed check and its numbers under the table, whose FAIL only said a question failed something. Signed-off-by: rlaope <[email protected]>
… and the drift skill The captured drift run is re-captured with the new failed-check line; the CI example adds the check only when costs.yaml exists; the skill pack is re-exported with tools/export_skills.py. Signed-off-by: rlaope <[email protected]>
Shadow traffic logs both versions on the same requests under one source_key. Pairing them by (source_key, question) removes request difficulty from the difference. calibration gains mcnemar_exact and paired_bootstrap (accuracy, ECE on fixed edges, Brier; resampled over pairs). drift.compare_paired pairs gold records, refuses duplicated keys, excludes pairs without gold on both sides or with conflicting labels, counts each exclusion, and refuses questions below 30 pairs. synth.generate_paired shares request keys, true answers and difficulty while leaving generate() byte-identical. Signed-off-by: rlaope <[email protected]>
The README quotes a captured run on examples/make-paired-log.py, where the ECE gate passes and the paired check finds the lower accuracy. test_readme_example reruns that block literally, and the documented features guard pins --paired in the README and llms.txt. Signed-off-by: rlaope <[email protected]>
Signed-off-by: rlaope <[email protected]> # Conflicts: # jeval/cli.py # jeval/drift.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--fail-on threshold-shift=X: fails a drift run when the cost-optimal threshold moves by more than X, in either direction. Without a cost matrix the check is refused with exit 1 instead of passing. The CI block now names each failed check and its numbers under the table.jeval drift --paired: compares two model versions on the same requests, matched by(source_key, question_key).--fail-onalone decides the exit code.jeval/calibration.py.jeval/synth.pygainsgenerate_pairedfor shadow-traffic logs. Existing generator output is byte-identical.examples/ci/drift.yml, the drift-gate skill (re-exported), andexamples/make-paired-log.py.Test plan
uv run pytest,ruffandmypyare clean.