runs: exploratory abstain/rag/equalized demos over shipped experiment datasets (deepseek)

[isaac-maya]

Runs that exercise the harness conditions in the companion PR (three opt-in flags: --abstain, --rag, --equalized). Every result file ships with its experiment dataset and a summary that records the model config, repo commit, date, and counting rules, so every number below is re-derivable from this PR alone.

What this PR is: 20 run files + 3 experiment datasets (data/experiments/), produced with deepseek-v4-flash / deepseek-v4-pro — open-weight, cheap, independent of the labs the benchmark measures (same rationale as the repo's self-hosted Qwen runs). Runs cost roughly $0.35 total at logged rate estimates; per-call logs stay local. The result files cannot move the leaderboard: the aggregator enumerates run-ids, and none of these are in its list.

What the runs show (all re-derivable from results/):

• Abstention on the full 34 (abstain34): the model abstained on 2 of 34 items — both cases where the description lacks the deciding parameter (a DSP whose crypto mass-market status is absent, an IMU whose spec does not settle the entry) — at confidence 0, with zero errors. On the same 12 items it guessed when not allowed to abstain (abstain_ab_closed_book: 2/12 exact, including a wrong guess on the item it later abstained on) and abstained on that item when it could (abstain_ab_abstain: 4/12 exact, 1 abstention). It states ~87% average confidence while being exactly right 40% of the time; the >=90 bin averages 91.5 at 40%. That gap is the finding: stated confidence is not calibrated, which is the precondition for trusting any confidence-based abstention policy.

• Retrieval, 12 items run twice per arm (rag_ab_closed_book_* vs rag_ab_retrieval_*): no net gain — 5/24 exact closed-book vs 4/24 with retrieval (grades 0.30/0.38 vs 0.36/0.33). The summaries record why: the gold entry's head is in the top-6 only 4 of 9 times on this set (hit_rate). Two mechanism examples in the repo's own error vocabulary: the 60 Hz thermal camera (exact closed-book, gold 6A003.b.4.b, rank fix(validate): widen the dataset leak check beyond the gold ECCN (description + item_name, EAR99 rows included) #1) moved to a wrong controlled entry (6A002) when 6A003's full text was injected — grounding inducing the error, the same direction as the repo's agentic finding — and the EAR99 Ethernet PHY (over-controlled to 5A991.b.4 closed-book) answered EAR99 with retrieval, n=1, an observation. This is the honest baseline a vector condition must beat.

• Equalized demo, 6 items x 2 (equalized_demo_a/b): under the matched-settings preset, flash and pro get the same number right (1/6 exact) and pro grades higher (0.48 vs 0.30 mean grade). The summaries record the per-model config check. Same-provider demo only — the summaries state it is not a cross-provider leaderboard.

Reproduce (from the repo root, after the companion PR merges):

python -m commoditybench.run_eval --dataset data/experiments/abstain_ab_12.jsonl --models deepseek-v4-flash --run-id abstain_ab_closed_book --temperature 0.0
python -m commoditybench.run_eval --dataset data/experiments/abstain_ab_12.jsonl --models deepseek-v4-flash --run-id abstain_ab_abstain --abstain --temperature 0.0
python -m commoditybench.run_eval --dataset data/questions.jsonl --models deepseek-v4-flash --run-id abstain34 --abstain --temperature 0.0
python -m commoditybench.run_eval --dataset data/experiments/rag_ab_12.jsonl --models deepseek-v4-flash --run-id rag_ab_retrieval_a --rag --temperature 0.0   # and the closed-book arm without --rag
python -m commoditybench.run_eval --dataset data/experiments/equalized_demo_6.jsonl --models deepseek-v4-flash deepseek-v4-pro --run-id equalized_demo_a --equalized

The experiment datasets mix published questions with the category-backfill candidates, chosen to concentrate the repo's known error modes (Ethernet-PHY over-control, 5A992.c mass-market, the frame-rate trap pair, scope-conditioned rows). Happy to drop any of these runs, re-run with the repo's own models/keys, or move them under a clearly labeled directory — the point of shipping them is that the claims carry their evidence.