Skip to content

feat(harness): three opt-in conditions (abstain scoring, BM25 RAG, equalized preset), defaults unchanged - #5

Open
isaac-maya wants to merge 1 commit into
IAPS-AI:mainfrom
isaac-maya:harness-optin-conditions
Open

isaac-maya wants to merge 1 commit into
IAPS-AI:mainfrom
isaac-maya:harness-optin-conditions

Conversation

@isaac-maya

Copy link
Copy Markdown

Ties to: the RAG stretch goal in the repo README, the "not equalized" caveat on the results site, and the zero-error standard behind the scoring.

What it does: three opt-in additions to the runner. Nothing changes unless the flag is passed.

  1. --abstain — lets a model say "I don't know" instead of guessing. Only verified abstentions count: non-empty response plus an explicit abstain: true with eccn: null. Empty or truncated responses are errors, never abstentions — the counting rule is written onto every run summary, and each row carries abstain / abstain_field / content_nonempty so every counted abstention can be re-checked against the raw response. Schema-strict providers (Anthropic/Gemini/OpenAI json_schema) additionally need COMMODITYBENCH_ABSTAIN_AWARE=1 set before start — the runner refuses to start without it for those models, because they could never emit an abstention under the 3-field schema.
  2. --rag — BM25 retrieval over the shipped CCL index (data/ccl/ccl_index.json). Pure Python, no new dependencies. Ranks on full entry text, injects a capped excerpt anchored on each entry's List of Items Controlled (full entries run to ~16k chars; the deciding thresholds live in the Items region). Every --rag summary records hit_rate — how often the gold entry's CCL head is in the retrieved top-k.
  3. --equalized — one inference preset for every model in a run, plus a deviation check that prints and records warnings when a model's effective config drifts. The run summary records the check and states it is not a cross-provider leaderboard.

Also in this PR: --log-calls (one JSON line per API call: exact prompts, raw response, tokens, cost estimate), and a config/date/repo-commit snapshot on every run summary. An analysis module (src/commoditybench/abstention.py) scores abstain runs in three buckets — answered / abstained / error — plus calibration (stated confidence vs actual accuracy, ECE) and an error taxonomy (four classes; errors and abstentions are never classification errors).

Why: three open threads in the repo. The zero-error standard leaves no way to refuse. RAG was scaffolding with no working retriever. And "not equalized" was a caveat nobody could check. All three are opt-in, so existing runs stay comparable.

They share one file (src/commoditybench/run_eval.py), so they come as one PR — code, tests, and design notes (11 files; the 5 source files are the review surface, the rest is tests and docs). The exploratory runs that exercise these conditions are in the companion PR, with every result file and experiment dataset shipped so all claims are re-derivable.

Tested: the 40 existing tests pass unchanged; 18 new tests cover the abstention accounting (error rows never count as abstentions, confidence 100 lands in the top calibration bin, every declared taxonomy class is reachable), the retriever context bound / Items-anchored rendering / hit-rate self-consistency, and the new runner guards (schema-strict detection, repo-relative summary paths). 58 total.

Open to feedback on any of the three conditions.

…ed preset (defaults unchanged), with tests and design docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant