feat: add wave 2 code benchmark adapters

benchmark/wave2-code/bigcodebench.md

@@ -0,0 +1,142 @@
+# BigCodeBench audit and integration note
+
+## Decision
+
+BigCodeBench now has an OpenBench adapter and an arm64 source-equivalent scorer
+image validated against all 148 canonical Hard records on Apple Silicon. It
+passes 145 records under the hardened no-network sandbox; the remaining three
+require live external downloads and therefore fail closed instead of being
+counted as model errors. The pinned official evaluator image is still
+`linux/amd64` only, so official-image parity remains blocked until it is
+smoke-tested on a usable native `linux/amd64` Docker host.
+
+## Canonical sources
+
+- Repository: `https://github.com/bigcode-project/bigcodebench.git`
+- Repository commit inspected: `09dd993f46c3fbf3a799465bb96d524edcb0b199`
+- Official package/data version used by upstream loader: `v0.1.4`
+- Full dataset: `bigcode/bigcodebench`, revision
+  `b74c0d0bf70d2c0bc459be537895cca163007f1a`, 1,140 tasks.
+- Hard dataset: `bigcode/bigcodebench-hard`, revision
+  `298d2cc7b96612e15e47313c3603ee124cee0c1f`, 148 tasks.
+- Official evaluator image: `bigcodebench/bigcodebench-evaluate` manifest
+  `sha256:1327bddf60be9bc241648c59e6060cac4ca50248a0588ab735cd0200b17cc8c2`
+  for `linux/amd64`.
+- License: Apache-2.0.
+
+## Implemented OpenBench surface
+
+- Registry ID: `bigcodebench`.
+- Parameters: `split="complete" | "instruct"`, `subset="full" | "hard"`,
+  `runtime="auto" | "official" | "arm64"`, optional `limit`, `epochs`, and
+  `total_timeout`. The source-equivalent arm64 runtime currently accepts only
+  `subset="hard"`; the unvalidated full subset fails at task construction.
+- Dataset loader: immutable Hugging Face revisions, with hidden execution fields
+  resolved only at score time.
+- Prompting: follows BigCodeBench's OpenAI/API chat backend wrapper by applying
+  the official instruction prefix and using the official complete-vs-instruct
+  prompt field.
+- Scoring: sandbox runner uses BigCodeBench's own `sanitize`, `trusted_check`,
+  and `untrusted_check`. Canonical solution timing is computed per task and used
+  to calibrate the generated solution timeout, matching the upstream evaluator.
+  If canonical calibration fails, the scorer raises an evaluation error rather
+  than silently assigning an incorrect model score.
+- Docker: `runtime="official"` pins the official `linux/amd64` evaluator image;
+  `runtime="arm64"` builds OpenBench's source-equivalent scorer image from the
+  pinned upstream source commit for Apple Silicon; `runtime="auto"` selects
+  arm64 on arm64/aarch64 hosts. Both compose files disable network, drop
+  capabilities, set no-new-privileges, and use tmpfs work directories.
+
+## Local validation
+
+Passed locally:
+
+```text
+source .venv/bin/activate && ruff check src/openbench/datasets/bigcodebench.py src/openbench/scorers/bigcodebench.py src/openbench/scorers/bigcodebench_runner.py src/openbench/evals/bigcodebench tests/test_bigcodebench.py tests/test_registry.py
+All checks passed!
+
+source .venv/bin/activate && pytest tests/test_bigcodebench.py tests/test_registry.py
+23 passed
+
+source .venv/bin/activate && mypy src/openbench/datasets/bigcodebench.py src/openbench/scorers/bigcodebench.py src/openbench/scorers/bigcodebench_runner.py src/openbench/evals/bigcodebench tests/test_bigcodebench.py
+Success: no issues found in 6 source files
+```
+
+Also passed dataset construction smoke for `bigcodebench(limit=1)` and the hard
+subset loader without printing benchmark prompts.
+
+Arm64 source-equivalent smoke passed locally:
+
+```text
+docker build --platform linux/arm64 -f src/openbench/evals/bigcodebench/Dockerfile.arm64 -t openbench-bigcodebench-arm64:dev src/openbench/evals/bigcodebench
+
+docker run --rm --network none --read-only --tmpfs /tmp:rw,nosuid,nodev,uid=1000,gid=1000,mode=0700,size=4294967296 --cap-drop ALL --security-opt no-new-privileges:true --pids-limit 256 --entrypoint python3 -v "$smoke_dir:/app:rw" openbench-bigcodebench-arm64:dev /app/runner.py /app/payload.json
+BigCodeBench/13
+{"passed": true, "status": "pass", ...}
+
+docker compose -f src/openbench/evals/bigcodebench/compose.arm64.yaml -p openbench-bcb-arm64-smoke up -d --build
+docker compose -f src/openbench/evals/bigcodebench/compose.arm64.yaml -p openbench-bcb-arm64-smoke exec -T default python3 -c "from bigcodebench.eval import untrusted_check; from bigcodebench.sanitize import sanitize; print('compose-imports-ok')"
+compose-imports-ok
+```
+
+The arm64 image intentionally omits BigCodeBench's generation-only API clients
+and vLLM dependency because OpenBench generates responses outside the scorer
+container. Its evaluator dependencies are fully resolved in
+`requirements-arm64.lock`; the Python base image, BigCodeBench source commit,
+toolchain, direct/transitive Python packages, and NLTK asset hashes are pinned.
+
+## Canonical Hard preflight
+
+The source-equivalent image was iterated against all 148 canonical Hard records.
+The machine-readable experiment log remains outside Git at
+`/tmp/openbench-bcb-arm64-validation/results.tsv`; prompts, tests, and canonical
+solutions are not included in the log.
+
+| Image state | Canonical pass rate | Wall time |
+| --- | ---: | ---: |
+| Minimal scorer baseline | 79/148 (53.38%) | 170.22 s |
+| First dependency layer | 133/148 (89.86%) | 607.98 s |
+| Second dependency/NLTK layer | 143/148 (96.62%) | 422.87 s |
+| Restored legacy imports | 144/148 (97.30%) | 437.43 s |
+| Hardened tmpfs caches | 145/148 (97.97%) | 500.89 s |
+
+The final three failures are `BigCodeBench/101`, `BigCodeBench/590`, and
+`BigCodeBench/1012`. Their canonical solutions access live resources hosted by
+CMU, Wikibooks, Google Drive, or learningcontainer. OpenBench keeps networking
+disabled for untrusted generated programs and does not replace these resources
+with invented fixtures. These samples therefore produce a canonical validation
+error and no model score.
+
+Blocked locally:
+
+```text
+docker run --platform linux/amd64 ... bigcodebench/bigcodebench-evaluate@sha256:1327bddf60be9bc241648c59e6060cac4ca50248a0588ab735cd0200b17cc8c2 ...
+qemu: uncaught target signal 11 (Segmentation fault) - core dumped
+```
+
+The host is `arm64` and Docker reports `linux/arm64`; the official evaluator is
+only available as `linux/amd64`. Even a synthetic tiny runner payload segfaulted
+under QEMU, so repeated local smoke attempts were stopped to avoid freezes.
+
+An x86_64 GPU instance (`ghub5090`) was also tried after explicit user approval.
+Docker was installed and started with bridge/iptables disabled, then retried with
+the `vfs` storage driver because the instance appears containerized. The daemon
+could start, but pulling/registering the official image failed with:
+
+```text
+failed to register layer: unshare: operation not permitted
+```
+
+Temporary Docker data and the smoke payload were removed from the GPU instance
+after the failed attempt. The `docker.io` package remains installed there because
+installation was explicitly approved for this validation attempt.
+
+## Remaining requirement
+
+For strict upstream-image parity, run the official Docker scorer smoke on a
+native `linux/amd64` host with Docker privileges sufficient for image extraction
+and container creation. A faithful full 148-task hardened run additionally needs
+immutable, audited fixtures for the three live-network records; enabling network
+for generated programs is not an acceptable fallback. Until both requirements
+are satisfied, BigCodeBench should be described as arm64 Hard 145/148 validated,
+not as full official-image parity.

benchmark/wave2-code/codeforces_elo.md

@@ -0,0 +1,40 @@
+# Codeforces ELO audit
+
+## Decision
+
+`Codeforces ELO` is recorded as unsupported because the cited Gemma 4 artifact
+publishes final rating values rather than a fixed benchmark protocol. No task
+window, contest/problem manifest, submission policy, judge environment, rating
+calculation, or canonical evaluation repository is linked from the model card.
+
+## Source evidence
+
+- Model card: `google/gemma-4-E4B-it`
+- Model-card revision inspected:
+  `ee0ef6023621cff504d758262d4e04895a5af4a2`
+- The benchmark table contains one `Codeforces ELO` row with final values for six
+  Gemma variants.
+- The card contains no other `Codeforces ELO` occurrence and supplies no
+  methodology note or source link for that row.
+
+An exact-name GitHub search found downstream catalog/ranking references, but no
+repository identified as the Gemma 4 evaluator. LiveOIBench and similarly named
+competitive-programming evaluations are distinct protocols and cannot be used
+as silent substitutes.
+
+## Missing reproducibility contract
+
+ELO is a derived rating, not a dataset-level metric. Reproducing it requires at
+least:
+
+- an immutable problem/contest/date window;
+- language, compiler, time, memory, and submission limits;
+- prompt and code-extraction rules;
+- sample count and generation settings;
+- the online/offline judge and hidden test assets;
+- the opponent/reference population and exact rating update formula.
+
+Without those pieces, an OpenBench `codeforces_elo` ID would assign the same name
+to an independently invented evaluation whose score is not comparable to the
+published row. The candidate remains blocked pending a canonical protocol from
+the benchmark publisher.

benchmark/wave2-code/contract.md

@@ -0,0 +1,56 @@
+# Wave 2 benchmark integration contract
+
+## Goal
+
+Integrate OJBench, TIR-Bench, Codeforces ELO, LiveBench, BigCodeBench, Spider,
+VITA-Bench, and MEGA into OpenBench without substituting similarly named data or
+approximating metrics that require unavailable model/provider capabilities.
+
+## Candidates
+
+The candidate list is frozen to the eight benchmark identities above. Each is
+treated as a separate integration candidate; no benchmark may borrow another's
+score or dataset identity.
+
+## Task matrix
+
+For every candidate:
+
+1. identify the canonical repository, release/dataset revision, license, prompt,
+   sampling settings, and scorer;
+2. record an immutable revision and checksums where practical;
+3. implement the complete public evaluation protocol or mark the candidate
+   unsupported with evidence;
+4. add registry metadata, unit tests, and a real dataset/task construction smoke;
+5. reuse a hardened Docker boundary whenever generated code, SQL, shell, or other
+   untrusted actions execute;
+6. run global lint, typing, unit tests, package build, and applicable Docker tests.
+
+## Metrics and fairness
+
+The primary integration metric is protocol completeness: `1` only when the
+canonical public protocol is runnable and tested, otherwise `0`. Secondary
+evidence records logical case count, task coverage, resource boundary, and known
+historical-score limitations. Every candidate receives the same source audit and
+validation gates; failures remain in `matrix.tsv` rather than being dropped.
+
+## Environment
+
+- Baseline commit: `7f11867` (`main`, after BFCL live merge)
+- Python: project `.venv`, managed by UV
+- Execution host: macOS/Docker Desktop; Linux execution images are digest-pinned
+- Upstream network artifacts may drift unless an immutable revision and digest
+  are recorded
+
+## Baseline
+
+At contract creation, none of the eight benchmark IDs exists in `src/` or the
+registry. The repository baseline passes 446 unit tests with two documented
+environment skips and four Docker sandbox integration tests.
+
+## Stopping condition
+
+Stop only when every candidate is either integrated and validated or explicitly
+blocked with source-backed evidence, then commit, open a pull request, and follow
+CI to completion. Model runs requiring unavailable paid credentials are logged as
+blocked and never represented as benchmark scores.

benchmark/wave2-code/matrix.tsv

@@ -0,0 +1,16 @@
+candidate	case	rep	metric	resource	status	notes
+ojbench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
+ojbench	official_protocol_audit	464_prompts_232_problems	0	dmoj+g++17+pypy3+git_lfs_testdata	unsupported	Official OJBench repo commit 5e94480b1e135b98855cf5bc81213c256aff5b17 and HF testdata HEAD 61cf9986f22c25d08e1657b03742124099c74353 expose 464 prompts with sha256 bcc8c94eb1fefb856355aa8b5a3e20cc0a2112f5436c5d83ab686edb417bce2c, but faithful judging requires DMOJ 4.1.0 at judge-server commit f098cd3a49a60186d1fadde5132329ec5f4f2213 plus g++17/PyPy3 and LFS problem zips; no hardened OpenBench Docker image has been validated for DMOJ under cap-drop/no-network, and OpenCompass only loads prompts without scoring.
+tir_bench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
+tir_bench	official_protocol_audit	1215_examples_13_tasks	0	agentic_image_harness+gpt4.1_extractor	unsupported	Official repository commit f79c7562b59e4f8142b0437fc725eb3ee1aec76c and HF dataset revision ae9976e81e86c4797fd0d28f9119393b37d9d006 publish data and post-generation scoring, but the released scripts require a GPT-4.1 answer extractor and do not publish the agentic image/tool generation harness underlying with-CI/without-CI runs.
+codeforces_elo	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
+codeforces_elo	model_card_protocol_audit	6_reported_model_ratings	0	missing_problem_window+judge+rating_formula	unsupported	Gemma 4 model card google/gemma-4-E4B-it revision ee0ef6023621cff504d758262d4e04895a5af4a2 publishes one Codeforces ELO row but no immutable problem manifest, generation protocol, judge environment, opponent population, rating formula, or canonical evaluator source; downstream competitive-programming benchmarks are not interchangeable.
+livebench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
+bigcodebench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
+bigcodebench	openbench_adapter	1140_full+148_hard	0	arm64_hard_145_of_148+x86_official	arm64_network_safe_validated_official_amd64_blocked	Implemented registry ID bigcodebench for official v0.1.4 complete/instruct and full/hard axes using HF dataset revisions b74c0d0bf70d2c0bc459be537895cca163007f1a and 298d2cc7b96612e15e47313c3603ee124cee0c1f plus pinned official amd64 evaluator image bigcodebench/bigcodebench-evaluate@sha256:1327bddf60be9bc241648c59e6060cac4ca50248a0588ab735cd0200b17cc8c2. The pinned source-equivalent arm64 image passes 145/148 canonical Hard records under network-none/read-only/cap-drop/no-new-privileges; records 101, 590, and 1012 require live external downloads and fail canonical calibration without producing model scores. Official amd64 image remains unvalidated locally because it segfaults under QEMU on arm64 and the x86_64 GPU instance disallows Docker layer registration/unshare.
+spider	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
+spider	official_protocol_audit	1034_dev	0	5.15GB_testsuite+missing_llm_prompt	unsupported	Official Spider repo commit b7b5b8c890cd30e35427348bb9eb8c6d1350ca7c and official test-suite-sql-eval commit e97acc546ecbee8fa27fa8dbf025ef61493a876c define data and test-suite execution accuracy, but no canonical LLM prompt/sampling protocol is published; the required test-suite DB archive is external Google Drive data sha256 9ec24ea8debc6bd04abfe137b5f1a739b5a8836f32c0464e4dfc94eb7f41da96, 1.2GB compressed and 5.15GB uncompressed, so OpenBench should not report a hand-prompted approximation.
+vita_bench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
+vita_bench	official_protocol_audit	400	0	3_llm_roles+external_judge	unsupported	Official repo commit 973756f4754873474e2931a404f68093df9ef4e2 and HF dataset HEAD 5ca6848c215cdffd5ef9bc704ddcb62ed74696f0 expose 100 cross-domain plus 300 single-domain tasks, but the public protocol requires a target agent LLM, LLM user simulator, and LLM trajectory evaluator configured through models.yaml; no faithful offline Inspect score can be produced without extra model credentials and judge authority.
+mega	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
+mega	official_protocol_audit	16_datasets_70_languages	0	openai+azure_translate+hf_endpoints	unsupported	Official repo commit 3e96bab146151942ed6a7bbe59c0364a78ebf94f is public MIT code, but the released framework is a script/notebook collection with provider-specific OpenAI/Azure/HF endpoint dependencies, hard-coded key/env requirements, PromptSource setup, and no immutable all-task main-results manifest suitable for a faithful provider-agnostic Inspect task.

benchmark/wave2-code/mega.md

@@ -0,0 +1,75 @@
+# MEGA audit
+
+## Decision
+
+MEGA is recorded as unsupported for this OpenBench wave. The public repository is
+available, but the released artifact is a collection of experiment scripts and
+notebooks for multilingual LLM evaluation rather than a single stable benchmark
+protocol that OpenBench can run provider-agnostically.
+
+## Canonical source
+
+- Repository: `https://github.com/microsoft/Multilingual-Evaluation-of-Generative-AI-MEGA.git`
+- Repository commit: `3e96bab146151942ed6a7bbe59c0364a78ebf94f`
+- License: MIT
+- Paper scope stated in README: 16 NLP datasets across 70 languages.
+
+## Official execution shape
+
+The README describes MEGA as a framework and documents XNLI as the concrete
+example. The repository then supplies task-specific shell scripts and notebooks.
+The current script inventory contains 21 `python -m mega...` invocations across
+these modules:
+
+- `mega.XLSUM`
+- `mega.analysis.contamination`
+- `mega.answer_cls`
+- `mega.eval_pawsx`
+- `mega.eval_qa_gptindex`
+- `mega.eval_qa_gptturbo`
+- `mega.eval_tag`
+- `mega.eval_xcopa`
+- `mega.eval_xnli`
+- `mega.eval_xstory_cloze`
+
+Those scripts encode per-task/per-language choices such as prompt names,
+few-shot counts, model names, validation-vs-test switches, translation modes,
+and metric output paths. There is no single main-results manifest that freezes
+all 16 datasets, languages, prompts, splits, few-shot selections, and model
+settings in one runnable protocol.
+
+## External service requirements
+
+The official setup requires API credentials in `keys/` for OpenAI and Bing
+Translator. The code also imports environment variables at module import time,
+including OpenAI endpoint settings, Hugging Face endpoint/key settings, and Bing
+Translator endpoint/key settings.
+
+`mega/models/completion_models.py` calls the legacy OpenAI completion/chat APIs
+directly, sleeps for rate limiting, and supports a fixed model list including
+Azure-style names such as `gpt-35-turbo`, `gpt-35-turbo-16k`, `gpt-4`, and
+`gpt-4-32k`, plus BLOOM/BLOOMZ through Hugging Face endpoints. That is not the
+same execution path as OpenBench's Inspect model abstraction.
+
+## Dependencies and artifacts
+
+The repository declares Python 3.7 compatibility and includes older unpinned or
+pinned dependencies such as `transformers==4.30.0`, `langchain==0.0.317`,
+`networkx==1.11`, `word2word==1.0.0`, `openai`, `datasets`, `evaluate`,
+`torch`, and a vendored PromptSource tree with 331 template files.
+
+The repository includes prior GPT-4 XLSUM artifacts under `gpt-4-all-lang-eval/`:
+35 prediction CSV files and `xlsum_gpt_4_metrics.csv` with SHA-256
+`a235258dc91cd5d219f8fde96376db1ee07f5de4a6dc0bb8ed9011477ef0b429`. These are
+historical result artifacts, not an evaluation dataset for arbitrary models.
+
+## OpenBench compatibility finding
+
+A faithful OpenBench integration would need a new MEGA manifest first: exact
+dataset revisions, language list per dataset, prompt template IDs, split policy,
+few-shot selection policy, translation-test policy, model sampling settings, and
+metric aggregation rules. It would also need a provider-agnostic rewrite of the
+OpenAI/Azure/HF-specific generation layer. Without those pieces, exposing a
+single `mega` registry ID would either cover only a hand-picked subset or report
+scores that are not comparable to the official MEGA experiments. The candidate
+is therefore blocked rather than approximated.