fix: validate OJBench across the full corpus

benchmark/wave2-code/codeforces_elo.md

@@ -16,6 +16,8 @@ calculation, or canonical evaluation repository is linked from the model card.
   Gemma variants.
 - The card contains no other `Codeforces ELO` occurrence and supplies no
   methodology note or source link for that row.
+- Gemma 4 Technical Report `arXiv:2607.02770v2` repeats the rating row but adds
+  no Codeforces methods section, task manifest, or evaluator reference.
 
 An exact-name GitHub search found downstream catalog/ranking references, but no
 repository identified as the Gemma 4 evaluator. LiveOIBench and similarly named

benchmark/wave2-code/matrix.tsv

@@ -1,9 +1,9 @@
 candidate	case	rep	metric	resource	status	notes
 ojbench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
 ojbench	official_protocol_audit	464_prompts_232_problems	0	dmoj+g++17+pypy3+git_lfs_testdata	unsupported	Official OJBench repo commit 5e94480b1e135b98855cf5bc81213c256aff5b17 and HF testdata HEAD 61cf9986f22c25d08e1657b03742124099c74353 expose 464 prompts with sha256 bcc8c94eb1fefb856355aa8b5a3e20cc0a2112f5436c5d83ab686edb417bce2c. The initial audit established that faithful judging required DMOJ 4.1.0 at judge-server commit f098cd3a49a60186d1fadde5132329ec5f4f2213 plus g++17/PyPy3 and LFS problem zips, while OpenCompass only loaded prompts without scoring; the subsequent alpha adapter row records the completed hardened-runtime work.
-ojbench	openbench_alpha_adapter	232_python+232_cpp_x8	0	lazy_pinned_testdata+dmoj_arm64	alpha_arm64_runtime_validated	Added ojbench_python and ojbench_cpp with eight official samples per problem, pinned prompts/testdata, and lazy per-problem downloads. The arm64 DMOJ image passed CPP17/PYPY3 self-tests and judged both languages over 20 real loj-2083 cases; an attempted hidden-test file read received IR. The container is network-none/read-only/no-new-privileges with bounded memory/PIDs, cap-drop ALL, and only DMOJ-required SYS_PTRACE restored. Full 7.85GB corpus validation and real model scoring remain future gates, so both IDs are alpha.
+ojbench	openbench_alpha_adapter	232_python+232_cpp_x8	0	7.51GB_archives+dmoj_arm64_bookworm	alpha_full_model_validated	Added ojbench_python and ojbench_cpp with eight official samples per problem, pinned prompts/testdata, and lazy per-problem downloads. The full 232-problem corpus produced 232/232 normal Python WA and 232/232 normal C++ WA sentinel verdicts with zero infrastructure errors; all 17 custom validator configurations passed. A full pinned RWKV-7 1.5B run then judged 3,712 unique generations with zero IE/Skip/exceptions: Python Pass@1 0.4310%, Pass@8 1.7241%; C++ Pass@1 0.2155%, Pass@8 0.8621%. Keep alpha until an Inspect-native or second provider stack completes the full protocol.
 tir_bench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
-tir_bench	official_protocol_audit	1215_examples_13_tasks	0	agentic_image_harness+gpt4.1_extractor	unsupported	Official repository commit f79c7562b59e4f8142b0437fc725eb3ee1aec76c and HF dataset revision ae9976e81e86c4797fd0d28f9119393b37d9d006 publish data and post-generation scoring, but the released scripts require a GPT-4.1 answer extractor and do not publish the agentic image/tool generation harness underlying with-CI/without-CI runs.
+tir_bench	official_protocol_audit	2591_rows_1215_scored	0	agentic_image_harness+live_extractor	partial_public_source	Official repository commit f79c7562b59e4f8142b0437fc725eb3ee1aec76c and HF revision 7fed373b469403c85755fb2629c79bcc74ad883b publish immutable data and post-generation scoring. The paper specifies GPT-4o extraction while current code defaults to GPT-4.1, and no complete provider-neutral with-CI harness is released. An explicitly named no-tools repo-GPT-4.1 variant is the bounded next slice.
 codeforces_elo	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
 codeforces_elo	model_card_protocol_audit	6_reported_model_ratings	0	missing_problem_window+judge+rating_formula	unsupported	Gemma 4 model card google/gemma-4-E4B-it revision ee0ef6023621cff504d758262d4e04895a5af4a2 publishes one Codeforces ELO row but no immutable problem manifest, generation protocol, judge environment, opponent population, rating formula, or canonical evaluator source; downstream competitive-programming benchmarks are not interchangeable.
 livebench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
@@ -12,6 +12,6 @@ bigcodebench	openbench_adapter	1140_full+148_hard	0	arm64_hard_145_of_148+x86_of
 spider	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
 spider	official_protocol_audit	1034_dev	0	5.15GB_testsuite+missing_llm_prompt	unsupported	Official Spider repo commit b7b5b8c890cd30e35427348bb9eb8c6d1350ca7c and official test-suite-sql-eval commit e97acc546ecbee8fa27fa8dbf025ef61493a876c define data and test-suite execution accuracy, but no canonical LLM prompt/sampling protocol is published; the required test-suite DB archive is external Google Drive data sha256 9ec24ea8debc6bd04abfe137b5f1a739b5a8836f32c0464e4dfc94eb7f41da96, 1.2GB compressed and 5.15GB uncompressed, so OpenBench should not report a hand-prompted approximation.
 vita_bench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
-vita_bench	official_protocol_audit	400	0	3_llm_roles+external_judge	unsupported	Official repo commit 973756f4754873474e2931a404f68093df9ef4e2 and HF dataset HEAD 5ca6848c215cdffd5ef9bc704ddcb62ed74696f0 expose 100 cross-domain plus 300 single-domain tasks, but the public protocol requires a target agent LLM, LLM user simulator, and LLM trajectory evaluator configured through models.yaml; no faithful offline Inspect score can be produced without extra model credentials and judge authority.
+vita_bench	official_protocol_audit	400_x4_bilingual	0	3_llm_roles+external_judge	live_implementation_queued	Current official repo commit 742e240855bf8686a0842360749d5ea970ea3987 and HF revision 8be56c8ca02d3d15cd3e8d27cc9162bc58502f01 expose the complete local 66-tool simulator plus Chinese and English variants of 400 tasks. An isolated Python 3.13/uv smoke loaded all eight 100-task language/domain manifests. Paper main results use four temperature-zero trials, GPT-4.1-2025-04-14 user simulation, and Claude 3.7 Sonnet trajectory evaluation. Prototype an explicit revisioned vitabench_v1_live ID with required auxiliary credentials and upstream differential tests; keep VitaBench 2.0 separate.
 mega	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
 mega	official_protocol_audit	16_datasets_70_languages	0	openai+azure_translate+hf_endpoints	unsupported	Official repo commit 3e96bab146151942ed6a7bbe59c0364a78ebf94f is public MIT code, but the released framework is a script/notebook collection with provider-specific OpenAI/Azure/HF endpoint dependencies, hard-coded key/env requirements, PromptSource setup, and no immutable all-task main-results manifest suitable for a faithful provider-agnostic Inspect task.

benchmark/wave2-code/ojbench.md

@@ -62,7 +62,8 @@ OpenBench integration.
 - Test archives and custom validators are downloaded lazily per problem into a
   host cache mounted read-only at `/problems`. A limited run therefore does not
   require downloading the complete 7.85 GB source repository.
-- The image installs OJBench commit
+- The image uses digest-pinned Python 3.11 Bookworm (GCC 12) and installs
+  OJBench commit
   `5e94480b1e135b98855cf5bc81213c256aff5b17` and DMOJ commit
   `f098cd3a49a60186d1fadde5132329ec5f4f2213`, with exact Python dependencies,
   C++17 `g++`, and PyPy3.
@@ -76,14 +77,58 @@ requires `SYS_PTRACE` to supervise its own child process, so that single
 capability is restored explicitly. Generated programs remain under DMOJ's
 seccomp and filesystem policies; `/problems` is not in their readable policy.
 
-## Validation and remaining gate
+## Validation
 
-On local arm64 Docker, both CPP17 and PYPY3 executor self-tests passed. Wrong
-Python and C++ submissions were judged against 20 real `loj-2083` cases and
-returned `WA`. A submission attempting to read the mounted hidden-test
+On local arm64 Docker, both CPP17 and PYPY3 executor self-tests passed. A
+synthetic one-case problem exercised AC, WA, CE, RTE/IR, TLE, and memory-pressure
+paths in both runtimes. A submission attempting to read the mounted hidden-test
 `init.yml` returned DMOJ `IR`, confirming the anti-oracle filesystem boundary.
 
-The complete 7.85 GB problem corpus has not yet been executed end to end, and
-no credentialed model run has been scored. The two registry IDs therefore
-remain alpha until broader custom-validator coverage and real Pass@1/Pass@8
-runs are complete.
+The complete pinned corpus was then downloaded: 232 problem directories and
+232 archives totalling 7,505,819,661 bytes. The preflight script evaluated a
+fast sentinel submission against every configuration with early stop after the
+first failure:
+
+- Python/PYPY3: 232/232 normal `WA` verdicts, zero infrastructure errors,
+  137.26 seconds wall time;
+- C++17: 232/232 normal `WA` verdicts, zero infrastructure errors, 247.54
+  seconds wall time;
+- all 17 problems using custom checkers, interactors, or output validators
+  compiled and returned normal verdicts.
+
+The first image used Debian Trixie/GCC 14 and exposed compilation failures in
+two upstream `testlib.h` validators. Switching to digest-pinned Bookworm/GCC 12
+restored all official validators without modifying benchmark artifacts.
+
+### Full real-model Pass@1/Pass@8 run
+
+A complete credential-free protocol run used
+`RWKV/RWKV7-1.5B-20260805` revision
+`bfb3a69a63e6681f729651c357f13ce0c774ea9c` on an RTX 5090. Generation used
+eight samples, seed 42, temperature 0.8, top-p 0.9, and at most 1,024 new
+tokens. The resulting 3,712-record JSONL contains exactly 1,856 generations per
+language and one record for every problem/language/epoch key; its SHA-256 is
+`93594d74033c147e452c44dde6e2ab3f288aa9e96b419833e6a520f7a7da9b8a`.
+
+Every generation was judged against the full pinned problem corpus. The
+validation harness stopped after the first non-AC case because the released
+metrics use only final AC; this preserves Pass@1/Pass@8 exactly while avoiding
+unnecessary hidden-case execution for already-failed programs. It emitted only
+IDs, verdicts, counts, and timing. The sanitized 3,712-row score file has
+SHA-256 `14895ae1b722e57e6b74f323b47511c5f8f22b312afeb91ce6f68b44a13b3ae9`
+and zero exceptions, `IE`, or `Skip` verdicts.
+
+| Track | Accepted samples | Problems solved at 8 | Pass@1 | Pass@8 |
+| --- | ---: | ---: | ---: | ---: |
+| Python | 8/1,856 | 4/232 | 0.4310% | 1.7241% |
+| C++ | 4/1,856 | 2/232 | 0.2155% | 0.8621% |
+
+Pass@1 is the official eight-sample estimator (accepted samples divided by
+1,856), not the score of an arbitrarily selected epoch. Pass@8 is the fraction
+of problems with at least one AC across the eight samples.
+
+The run closes the full-corpus generation/judging/reducer gate. The registry IDs
+remain alpha because generation used a direct pinned Transformers model rather
+than an Inspect-native provider and only one model stack has completed the full
+protocol. The score validates the implementation; it is not a leaderboard
+quality claim.

benchmark/wave2-code/tir_bench.md

@@ -15,9 +15,10 @@ a separately configured GPT-4.1 judge.
 - Repository commit inspected:
   `f79c7562b59e4f8142b0437fc725eb3ee1aec76c`
 - Dataset: `Agents-X/TIR-Bench`
-- Dataset revision: `ae9976e81e86c4797fd0d28f9119393b37d9d006`
+- Dataset revision: `7fed373b469403c85755fb2629c79bcc74ad883b`
 - Dataset license: Apache-2.0.
-- Published scope: 1,215 examples across 13 image-reasoning tasks.
+- Current Hub scope: 2,591 rows across 15 configurations; the released
+  extraction path asserts 1,215 scored responses across 13 task families.
 
 The repository itself does not include a license file. The Apache-2.0 label
 above comes from the immutable Hugging Face dataset metadata and should not be
@@ -35,6 +36,10 @@ The repository provides two post-generation stages:
    jigsaw-position accuracy, OCR substring checks, Levenshtein normalization,
    and `math_verify` fallbacks.
 
+The paper says GPT-4o was used for extraction, while the current repository
+defaults to GPT-4.1. These are distinct published protocol variants and should
+not share one score identity without a differential comparison.
+
 The scripts expect model responses to exist already. They do not define how the
 evaluated model receives images, creates or invokes image-processing tools,
 iterates over intermediate images, limits tool calls, or converts that trajectory
@@ -44,10 +49,8 @@ runtime.
 
 ## OpenBench compatibility finding
 
-A faithful integration needs two missing protocol components: the canonical
-agent/tool generation harness and explicit authority/configuration for the
-GPT-4.1 extraction judge. A static VQA prompt would measure a different task,
-while substituting a hand-authored deterministic extractor would change the
-published metric. OpenBench therefore records the public data and scorer but
-does not expose a `tir_bench` task until those choices can be reproduced or made
-explicit in the benchmark identity.
+A faithful full integration still needs the canonical agent/tool generation
+harness. A bounded next slice is an explicitly named no-tools variant using the
+dataset's published instructions and the repository GPT-4.1 extractor; it must
+not be presented as the paper's with-CI aggregate. Substituting a hand-authored
+deterministic extractor would change the published metric.

benchmark/wave2-code/validate_ojbench_arm64.py

@@ -0,0 +1,97 @@
+"""Preflight OJBench problem configurations inside the DMOJ scorer image.
+
+The payload contains only problem identifiers and a language. This diagnostic
+uses a deliberately wrong submission, stops execution after the first failed
+case, and emits only identifiers, verdicts, counts, error types, and timing.
+Prompts, test contents, per-case output, and judge feedback are never printed.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+import ojbench  # type: ignore[import-not-found]
+from ojbench.judger import judge  # type: ignore[import-not-found]
+
+RESULT_PREFIX = "OPENBENCH_RESULT\t"
+SUMMARY_PREFIX = "OPENBENCH_SUMMARY\t"
+SOURCES = {
+    "python": ("PYPY3", "print(0)"),
+    "cpp": ("CPP17", "#include <iostream>\nint main(){std::cout << 0;}"),
+}
+
+
+def validate_problem(problem_id: str, language: str) -> dict[str, Any]:
+    """Load one problem and execute a fast sentinel submission."""
+
+    runtime, source = SOURCES[language]
+    started = time.monotonic()
+    try:
+        verdict, results = judge(
+            problem_id=problem_id,
+            time_limit=10,
+            memory_limit=1024 * 1024,
+            language=runtime,
+            source=source,
+            stop_when_fail=True,
+            use_tqdm=False,
+        )
+        return {
+            "problem_id": problem_id,
+            "language": language,
+            "verdict": verdict,
+            "tests": len(results),
+            "wall_seconds": round(time.monotonic() - started, 6),
+            "error_type": None,
+        }
+    except Exception as exc:  # noqa: BLE001 - diagnostic boundary
+        return {
+            "problem_id": problem_id,
+            "language": language,
+            "verdict": None,
+            "tests": 0,
+            "wall_seconds": round(time.monotonic() - started, 6),
+            "error_type": type(exc).__name__,
+        }
+
+
+def main() -> None:
+    """Validate every requested problem after one DMOJ initialization."""
+
+    payload = json.loads(Path(sys.argv[1]).read_text())
+    language = str(payload["language"])
+    if language not in SOURCES:
+        raise ValueError("language must be 'python' or 'cpp'")
+    problem_ids = [str(problem_id) for problem_id in payload["problem_ids"]]
+
+    ojbench.init(
+        problem_dirs=[Path("/problems/NOI"), Path("/problems/ICPC")],
+        config_path=Path(os.environ["OJBENCH_CONFIG_PATH"]),
+        runtime_path=Path(os.environ["OJBENCH_RUNTIME_PATH"]),
+        compile_lock_path=Path("/workspace/compile.lock"),
+    )
+
+    started = time.monotonic()
+    infrastructure_errors = 0
+    for problem_id in problem_ids:
+        result = validate_problem(problem_id, language)
+        if result["error_type"] is not None or result["verdict"] in {"IE", "Skip"}:
+            infrastructure_errors += 1
+        print(f"{RESULT_PREFIX}{json.dumps(result, sort_keys=True)}", flush=True)
+
+    summary = {
+        "language": language,
+        "problems": len(problem_ids),
+        "infrastructure_errors": infrastructure_errors,
+        "wall_seconds": round(time.monotonic() - started, 6),
+    }
+    print(f"{SUMMARY_PREFIX}{json.dumps(summary, sort_keys=True)}", flush=True)
+
+
+if __name__ == "__main__":
+    main()