feat(harness): three opt-in conditions (abstain scoring, BM25 RAG, equalized preset), defaults unchanged - #5
Open
isaac-maya wants to merge 1 commit into
Conversation
…ed preset (defaults unchanged), with tests and design docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ties to: the RAG stretch goal in the repo README, the "not equalized" caveat on the results site, and the zero-error standard behind the scoring.
What it does: three opt-in additions to the runner. Nothing changes unless the flag is passed.
--abstain— lets a model say "I don't know" instead of guessing. Only verified abstentions count: non-empty response plus an explicitabstain: truewitheccn: null. Empty or truncated responses are errors, never abstentions — the counting rule is written onto every run summary, and each row carriesabstain/abstain_field/content_nonemptyso every counted abstention can be re-checked against the raw response. Schema-strict providers (Anthropic/Gemini/OpenAI json_schema) additionally needCOMMODITYBENCH_ABSTAIN_AWARE=1set before start — the runner refuses to start without it for those models, because they could never emit an abstention under the 3-field schema.--rag— BM25 retrieval over the shipped CCL index (data/ccl/ccl_index.json). Pure Python, no new dependencies. Ranks on full entry text, injects a capped excerpt anchored on each entry's List of Items Controlled (full entries run to ~16k chars; the deciding thresholds live in the Items region). Every--ragsummary recordshit_rate— how often the gold entry's CCL head is in the retrieved top-k.--equalized— one inference preset for every model in a run, plus a deviation check that prints and records warnings when a model's effective config drifts. The run summary records the check and states it is not a cross-provider leaderboard.Also in this PR:
--log-calls(one JSON line per API call: exact prompts, raw response, tokens, cost estimate), and a config/date/repo-commit snapshot on every run summary. An analysis module (src/commoditybench/abstention.py) scores abstain runs in three buckets — answered / abstained / error — plus calibration (stated confidence vs actual accuracy, ECE) and an error taxonomy (four classes; errors and abstentions are never classification errors).Why: three open threads in the repo. The zero-error standard leaves no way to refuse. RAG was scaffolding with no working retriever. And "not equalized" was a caveat nobody could check. All three are opt-in, so existing runs stay comparable.
They share one file (
src/commoditybench/run_eval.py), so they come as one PR — code, tests, and design notes (11 files; the 5 source files are the review surface, the rest is tests and docs). The exploratory runs that exercise these conditions are in the companion PR, with every result file and experiment dataset shipped so all claims are re-derivable.Tested: the 40 existing tests pass unchanged; 18 new tests cover the abstention accounting (error rows never count as abstentions, confidence 100 lands in the top calibration bin, every declared taxonomy class is reachable), the retriever context bound / Items-anchored rendering / hit-rate self-consistency, and the new runner guards (schema-strict detection, repo-relative summary paths). 58 total.
Open to feedback on any of the three conditions.