Feat/verify and decontaminate - #2
Merged
Merged
Conversation
Fix correctness bugs found while auditing the SDK-facing commands: - decontaminate: a benchmark prompt shorter than the n-gram size that appeared verbatim inside a longer training row was silently not flagged. Index short benchmark texts at their own length and match same-size windows so embedded contamination is caught. - verify --kind math: whole-number answers now require exact equality; the relative tolerance accepted off-by-one errors once the answer magnitude reached ~1e6. - verify --kind code: read the pass tally from a unique sentinel and the last match so a candidate can't spoof its own result; drop empty test entries instead of counting them as passes. - decontaminate: an n-gram size below 1 now fails instead of flagging and deleting every row. Security: verify --kind code runs dataset code in a subprocess, which is not a sandbox. Correct the misleading docstring and print a warning before running so it is not treated as safe on untrusted data. Tests: add coverage for run.py, cli.py, and publish.py via the client= injection hooks (no network or real SDK), plus a regression test for each fix above. Suite goes from 35 to 80 passing, ruff clean. Cut v0.2.0: move doctor/suggest out of "in progress" (both are shipped and tested), bump the package version, fix the [notebooks] -> [all] extra, refresh the MAP command table, and correct the quickstart Python version and pip name. Remove the no-op _tri() wrapper.
A1VARA5
force-pushed
the
feat/verify-and-decontaminate
branch
from
July 17, 2026 18:12
a480f18 to
afb472b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request
What this changes
A short summary of the change and why.
Type
Checklist
Anything reviewers should know
Context, open questions, or follow ups.