diff --git a/benchmark/wave2-code/codeforces_elo.md b/benchmark/wave2-code/codeforces_elo.md index 8f2e92f7..e2518fbf 100644 --- a/benchmark/wave2-code/codeforces_elo.md +++ b/benchmark/wave2-code/codeforces_elo.md @@ -16,6 +16,8 @@ calculation, or canonical evaluation repository is linked from the model card. Gemma variants. - The card contains no other `Codeforces ELO` occurrence and supplies no methodology note or source link for that row. +- Gemma 4 Technical Report `arXiv:2607.02770v2` repeats the rating row but adds + no Codeforces methods section, task manifest, or evaluator reference. An exact-name GitHub search found downstream catalog/ranking references, but no repository identified as the Gemma 4 evaluator. LiveOIBench and similarly named diff --git a/benchmark/wave2-code/matrix.tsv b/benchmark/wave2-code/matrix.tsv index 40214ee6..c913a045 100644 --- a/benchmark/wave2-code/matrix.tsv +++ b/benchmark/wave2-code/matrix.tsv @@ -1,9 +1,9 @@ candidate case rep metric resource status notes ojbench registry_baseline 1 0 0 unsupported No implementation or registry ID at baseline 7f11867 ojbench official_protocol_audit 464_prompts_232_problems 0 dmoj+g++17+pypy3+git_lfs_testdata unsupported Official OJBench repo commit 5e94480b1e135b98855cf5bc81213c256aff5b17 and HF testdata HEAD 61cf9986f22c25d08e1657b03742124099c74353 expose 464 prompts with sha256 bcc8c94eb1fefb856355aa8b5a3e20cc0a2112f5436c5d83ab686edb417bce2c. The initial audit established that faithful judging required DMOJ 4.1.0 at judge-server commit f098cd3a49a60186d1fadde5132329ec5f4f2213 plus g++17/PyPy3 and LFS problem zips, while OpenCompass only loaded prompts without scoring; the subsequent alpha adapter row records the completed hardened-runtime work. -ojbench openbench_alpha_adapter 232_python+232_cpp_x8 0 lazy_pinned_testdata+dmoj_arm64 alpha_arm64_runtime_validated Added ojbench_python and ojbench_cpp with eight official samples per problem, pinned prompts/testdata, and lazy per-problem downloads. The arm64 DMOJ image passed CPP17/PYPY3 self-tests and judged both languages over 20 real loj-2083 cases; an attempted hidden-test file read received IR. The container is network-none/read-only/no-new-privileges with bounded memory/PIDs, cap-drop ALL, and only DMOJ-required SYS_PTRACE restored. Full 7.85GB corpus validation and real model scoring remain future gates, so both IDs are alpha. +ojbench openbench_alpha_adapter 232_python+232_cpp_x8 0 7.51GB_archives+dmoj_arm64_bookworm alpha_full_model_validated Added ojbench_python and ojbench_cpp with eight official samples per problem, pinned prompts/testdata, and lazy per-problem downloads. The full 232-problem corpus produced 232/232 normal Python WA and 232/232 normal C++ WA sentinel verdicts with zero infrastructure errors; all 17 custom validator configurations passed. A full pinned RWKV-7 1.5B run then judged 3,712 unique generations with zero IE/Skip/exceptions: Python Pass@1 0.4310%, Pass@8 1.7241%; C++ Pass@1 0.2155%, Pass@8 0.8621%. Keep alpha until an Inspect-native or second provider stack completes the full protocol. tir_bench registry_baseline 1 0 0 unsupported No implementation or registry ID at baseline 7f11867 -tir_bench official_protocol_audit 1215_examples_13_tasks 0 agentic_image_harness+gpt4.1_extractor unsupported Official repository commit f79c7562b59e4f8142b0437fc725eb3ee1aec76c and HF dataset revision ae9976e81e86c4797fd0d28f9119393b37d9d006 publish data and post-generation scoring, but the released scripts require a GPT-4.1 answer extractor and do not publish the agentic image/tool generation harness underlying with-CI/without-CI runs. +tir_bench official_protocol_audit 2591_rows_1215_scored 0 agentic_image_harness+live_extractor partial_public_source Official repository commit f79c7562b59e4f8142b0437fc725eb3ee1aec76c and HF revision 7fed373b469403c85755fb2629c79bcc74ad883b publish immutable data and post-generation scoring. The paper specifies GPT-4o extraction while current code defaults to GPT-4.1, and no complete provider-neutral with-CI harness is released. An explicitly named no-tools repo-GPT-4.1 variant is the bounded next slice. codeforces_elo registry_baseline 1 0 0 unsupported No implementation or registry ID at baseline 7f11867 codeforces_elo model_card_protocol_audit 6_reported_model_ratings 0 missing_problem_window+judge+rating_formula unsupported Gemma 4 model card google/gemma-4-E4B-it revision ee0ef6023621cff504d758262d4e04895a5af4a2 publishes one Codeforces ELO row but no immutable problem manifest, generation protocol, judge environment, opponent population, rating formula, or canonical evaluator source; downstream competitive-programming benchmarks are not interchangeable. livebench registry_baseline 1 0 0 unsupported No implementation or registry ID at baseline 7f11867 @@ -12,6 +12,6 @@ bigcodebench openbench_adapter 1140_full+148_hard 0 arm64_hard_145_of_148+x86_of spider registry_baseline 1 0 0 unsupported No implementation or registry ID at baseline 7f11867 spider official_protocol_audit 1034_dev 0 5.15GB_testsuite+missing_llm_prompt unsupported Official Spider repo commit b7b5b8c890cd30e35427348bb9eb8c6d1350ca7c and official test-suite-sql-eval commit e97acc546ecbee8fa27fa8dbf025ef61493a876c define data and test-suite execution accuracy, but no canonical LLM prompt/sampling protocol is published; the required test-suite DB archive is external Google Drive data sha256 9ec24ea8debc6bd04abfe137b5f1a739b5a8836f32c0464e4dfc94eb7f41da96, 1.2GB compressed and 5.15GB uncompressed, so OpenBench should not report a hand-prompted approximation. vita_bench registry_baseline 1 0 0 unsupported No implementation or registry ID at baseline 7f11867 -vita_bench official_protocol_audit 400 0 3_llm_roles+external_judge unsupported Official repo commit 973756f4754873474e2931a404f68093df9ef4e2 and HF dataset HEAD 5ca6848c215cdffd5ef9bc704ddcb62ed74696f0 expose 100 cross-domain plus 300 single-domain tasks, but the public protocol requires a target agent LLM, LLM user simulator, and LLM trajectory evaluator configured through models.yaml; no faithful offline Inspect score can be produced without extra model credentials and judge authority. +vita_bench official_protocol_audit 400_x4_bilingual 0 3_llm_roles+external_judge live_implementation_queued Current official repo commit 742e240855bf8686a0842360749d5ea970ea3987 and HF revision 8be56c8ca02d3d15cd3e8d27cc9162bc58502f01 expose the complete local 66-tool simulator plus Chinese and English variants of 400 tasks. An isolated Python 3.13/uv smoke loaded all eight 100-task language/domain manifests. Paper main results use four temperature-zero trials, GPT-4.1-2025-04-14 user simulation, and Claude 3.7 Sonnet trajectory evaluation. Prototype an explicit revisioned vitabench_v1_live ID with required auxiliary credentials and upstream differential tests; keep VitaBench 2.0 separate. mega registry_baseline 1 0 0 unsupported No implementation or registry ID at baseline 7f11867 mega official_protocol_audit 16_datasets_70_languages 0 openai+azure_translate+hf_endpoints unsupported Official repo commit 3e96bab146151942ed6a7bbe59c0364a78ebf94f is public MIT code, but the released framework is a script/notebook collection with provider-specific OpenAI/Azure/HF endpoint dependencies, hard-coded key/env requirements, PromptSource setup, and no immutable all-task main-results manifest suitable for a faithful provider-agnostic Inspect task. diff --git a/benchmark/wave2-code/ojbench.md b/benchmark/wave2-code/ojbench.md index c811c2cd..5f45f30e 100644 --- a/benchmark/wave2-code/ojbench.md +++ b/benchmark/wave2-code/ojbench.md @@ -62,7 +62,8 @@ OpenBench integration. - Test archives and custom validators are downloaded lazily per problem into a host cache mounted read-only at `/problems`. A limited run therefore does not require downloading the complete 7.85 GB source repository. -- The image installs OJBench commit +- The image uses digest-pinned Python 3.11 Bookworm (GCC 12) and installs + OJBench commit `5e94480b1e135b98855cf5bc81213c256aff5b17` and DMOJ commit `f098cd3a49a60186d1fadde5132329ec5f4f2213`, with exact Python dependencies, C++17 `g++`, and PyPy3. @@ -76,14 +77,58 @@ requires `SYS_PTRACE` to supervise its own child process, so that single capability is restored explicitly. Generated programs remain under DMOJ's seccomp and filesystem policies; `/problems` is not in their readable policy. -## Validation and remaining gate +## Validation -On local arm64 Docker, both CPP17 and PYPY3 executor self-tests passed. Wrong -Python and C++ submissions were judged against 20 real `loj-2083` cases and -returned `WA`. A submission attempting to read the mounted hidden-test +On local arm64 Docker, both CPP17 and PYPY3 executor self-tests passed. A +synthetic one-case problem exercised AC, WA, CE, RTE/IR, TLE, and memory-pressure +paths in both runtimes. A submission attempting to read the mounted hidden-test `init.yml` returned DMOJ `IR`, confirming the anti-oracle filesystem boundary. -The complete 7.85 GB problem corpus has not yet been executed end to end, and -no credentialed model run has been scored. The two registry IDs therefore -remain alpha until broader custom-validator coverage and real Pass@1/Pass@8 -runs are complete. +The complete pinned corpus was then downloaded: 232 problem directories and +232 archives totalling 7,505,819,661 bytes. The preflight script evaluated a +fast sentinel submission against every configuration with early stop after the +first failure: + +- Python/PYPY3: 232/232 normal `WA` verdicts, zero infrastructure errors, + 137.26 seconds wall time; +- C++17: 232/232 normal `WA` verdicts, zero infrastructure errors, 247.54 + seconds wall time; +- all 17 problems using custom checkers, interactors, or output validators + compiled and returned normal verdicts. + +The first image used Debian Trixie/GCC 14 and exposed compilation failures in +two upstream `testlib.h` validators. Switching to digest-pinned Bookworm/GCC 12 +restored all official validators without modifying benchmark artifacts. + +### Full real-model Pass@1/Pass@8 run + +A complete credential-free protocol run used +`RWKV/RWKV7-1.5B-20260805` revision +`bfb3a69a63e6681f729651c357f13ce0c774ea9c` on an RTX 5090. Generation used +eight samples, seed 42, temperature 0.8, top-p 0.9, and at most 1,024 new +tokens. The resulting 3,712-record JSONL contains exactly 1,856 generations per +language and one record for every problem/language/epoch key; its SHA-256 is +`93594d74033c147e452c44dde6e2ab3f288aa9e96b419833e6a520f7a7da9b8a`. + +Every generation was judged against the full pinned problem corpus. The +validation harness stopped after the first non-AC case because the released +metrics use only final AC; this preserves Pass@1/Pass@8 exactly while avoiding +unnecessary hidden-case execution for already-failed programs. It emitted only +IDs, verdicts, counts, and timing. The sanitized 3,712-row score file has +SHA-256 `14895ae1b722e57e6b74f323b47511c5f8f22b312afeb91ce6f68b44a13b3ae9` +and zero exceptions, `IE`, or `Skip` verdicts. + +| Track | Accepted samples | Problems solved at 8 | Pass@1 | Pass@8 | +| --- | ---: | ---: | ---: | ---: | +| Python | 8/1,856 | 4/232 | 0.4310% | 1.7241% | +| C++ | 4/1,856 | 2/232 | 0.2155% | 0.8621% | + +Pass@1 is the official eight-sample estimator (accepted samples divided by +1,856), not the score of an arbitrarily selected epoch. Pass@8 is the fraction +of problems with at least one AC across the eight samples. + +The run closes the full-corpus generation/judging/reducer gate. The registry IDs +remain alpha because generation used a direct pinned Transformers model rather +than an Inspect-native provider and only one model stack has completed the full +protocol. The score validates the implementation; it is not a leaderboard +quality claim. diff --git a/benchmark/wave2-code/tir_bench.md b/benchmark/wave2-code/tir_bench.md index 4269de64..1b0d3b4e 100644 --- a/benchmark/wave2-code/tir_bench.md +++ b/benchmark/wave2-code/tir_bench.md @@ -15,9 +15,10 @@ a separately configured GPT-4.1 judge. - Repository commit inspected: `f79c7562b59e4f8142b0437fc725eb3ee1aec76c` - Dataset: `Agents-X/TIR-Bench` -- Dataset revision: `ae9976e81e86c4797fd0d28f9119393b37d9d006` +- Dataset revision: `7fed373b469403c85755fb2629c79bcc74ad883b` - Dataset license: Apache-2.0. -- Published scope: 1,215 examples across 13 image-reasoning tasks. +- Current Hub scope: 2,591 rows across 15 configurations; the released + extraction path asserts 1,215 scored responses across 13 task families. The repository itself does not include a license file. The Apache-2.0 label above comes from the immutable Hugging Face dataset metadata and should not be @@ -35,6 +36,10 @@ The repository provides two post-generation stages: jigsaw-position accuracy, OCR substring checks, Levenshtein normalization, and `math_verify` fallbacks. +The paper says GPT-4o was used for extraction, while the current repository +defaults to GPT-4.1. These are distinct published protocol variants and should +not share one score identity without a differential comparison. + The scripts expect model responses to exist already. They do not define how the evaluated model receives images, creates or invokes image-processing tools, iterates over intermediate images, limits tool calls, or converts that trajectory @@ -44,10 +49,8 @@ runtime. ## OpenBench compatibility finding -A faithful integration needs two missing protocol components: the canonical -agent/tool generation harness and explicit authority/configuration for the -GPT-4.1 extraction judge. A static VQA prompt would measure a different task, -while substituting a hand-authored deterministic extractor would change the -published metric. OpenBench therefore records the public data and scorer but -does not expose a `tir_bench` task until those choices can be reproduced or made -explicit in the benchmark identity. +A faithful full integration still needs the canonical agent/tool generation +harness. A bounded next slice is an explicitly named no-tools variant using the +dataset's published instructions and the repository GPT-4.1 extractor; it must +not be presented as the paper's with-CI aggregate. Substituting a hand-authored +deterministic extractor would change the published metric. diff --git a/benchmark/wave2-code/validate_ojbench_arm64.py b/benchmark/wave2-code/validate_ojbench_arm64.py new file mode 100644 index 00000000..3dd1654c --- /dev/null +++ b/benchmark/wave2-code/validate_ojbench_arm64.py @@ -0,0 +1,97 @@ +"""Preflight OJBench problem configurations inside the DMOJ scorer image. + +The payload contains only problem identifiers and a language. This diagnostic +uses a deliberately wrong submission, stops execution after the first failed +case, and emits only identifiers, verdicts, counts, error types, and timing. +Prompts, test contents, per-case output, and judge feedback are never printed. +""" + +from __future__ import annotations + +import json +import os +import sys +import time +from pathlib import Path +from typing import Any + +import ojbench # type: ignore[import-not-found] +from ojbench.judger import judge # type: ignore[import-not-found] + +RESULT_PREFIX = "OPENBENCH_RESULT\t" +SUMMARY_PREFIX = "OPENBENCH_SUMMARY\t" +SOURCES = { + "python": ("PYPY3", "print(0)"), + "cpp": ("CPP17", "#include \nint main(){std::cout << 0;}"), +} + + +def validate_problem(problem_id: str, language: str) -> dict[str, Any]: + """Load one problem and execute a fast sentinel submission.""" + + runtime, source = SOURCES[language] + started = time.monotonic() + try: + verdict, results = judge( + problem_id=problem_id, + time_limit=10, + memory_limit=1024 * 1024, + language=runtime, + source=source, + stop_when_fail=True, + use_tqdm=False, + ) + return { + "problem_id": problem_id, + "language": language, + "verdict": verdict, + "tests": len(results), + "wall_seconds": round(time.monotonic() - started, 6), + "error_type": None, + } + except Exception as exc: # noqa: BLE001 - diagnostic boundary + return { + "problem_id": problem_id, + "language": language, + "verdict": None, + "tests": 0, + "wall_seconds": round(time.monotonic() - started, 6), + "error_type": type(exc).__name__, + } + + +def main() -> None: + """Validate every requested problem after one DMOJ initialization.""" + + payload = json.loads(Path(sys.argv[1]).read_text()) + language = str(payload["language"]) + if language not in SOURCES: + raise ValueError("language must be 'python' or 'cpp'") + problem_ids = [str(problem_id) for problem_id in payload["problem_ids"]] + + ojbench.init( + problem_dirs=[Path("/problems/NOI"), Path("/problems/ICPC")], + config_path=Path(os.environ["OJBENCH_CONFIG_PATH"]), + runtime_path=Path(os.environ["OJBENCH_RUNTIME_PATH"]), + compile_lock_path=Path("/workspace/compile.lock"), + ) + + started = time.monotonic() + infrastructure_errors = 0 + for problem_id in problem_ids: + result = validate_problem(problem_id, language) + if result["error_type"] is not None or result["verdict"] in {"IE", "Skip"}: + infrastructure_errors += 1 + print(f"{RESULT_PREFIX}{json.dumps(result, sort_keys=True)}", flush=True) + + summary = { + "language": language, + "problems": len(problem_ids), + "infrastructure_errors": infrastructure_errors, + "wall_seconds": round(time.monotonic() - started, 6), + } + print(f"{SUMMARY_PREFIX}{json.dumps(summary, sort_keys=True)}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/benchmark/wave2-code/vita_bench.md b/benchmark/wave2-code/vita_bench.md index fcaa9270..65ab7709 100644 --- a/benchmark/wave2-code/vita_bench.md +++ b/benchmark/wave2-code/vita_bench.md @@ -2,30 +2,38 @@ ## Decision -VITA-Bench is recorded as unsupported for this OpenBench wave. The public -benchmark is available, but the official protocol is not a single-model offline -task: it requires a model under test, a separate LLM user simulator, and a -separate LLM trajectory evaluator. OpenBench should not report a VITA score by -substituting a deterministic scorer or by silently choosing auxiliary LLMs. +VitaBench 1.0 is actionable as an explicit live benchmark candidate. The +complete simulated environment, 400 tasks, user simulator, prompts, and +trajectory evaluator are public and pinnable. It is not a single-model offline +task: a faithful run requires the model under test, a fixed auxiliary user LLM, +and a fixed auxiliary evaluator LLM. OpenBench should therefore prototype +`vitabench_v1_live` and withhold the unqualified alias until credentialed +differential runs against the upstream CLI pass. ## Canonical sources -- Repository: `https://github.com/meituan/vitabench.git` -- Repository commit: `973756f4754873474e2931a404f68093df9ef4e2` +- Repository: `https://github.com/meituan-longcat/vitabench.git` +- Repository commit: `742e240855bf8686a0842360749d5ea970ea3987` - Dataset: `https://huggingface.co/datasets/meituan-longcat/VitaBench` -- Dataset HEAD: `5ca6848c215cdffd5ef9bc704ddcb62ed74696f0` +- Dataset revision: `8be56c8ca02d3d15cd3e8d27cc9162bc58502f01` +- Paper: `https://arxiv.org/abs/2509.26490v2` - License: MIT ## Public task assets -The official repository includes four task files under `data/vita/domains/`: +The official repository includes Chinese and English variants of four task +manifests under `data/vita/domains/`: -| Domain | Tasks | SHA-256 | Bytes | +| Domain/language | Tasks | SHA-256 | Bytes | | --- | ---: | --- | ---: | -| `cross_domain` | 100 | `3d662cd36efae511e256842e81d54b97f399152be9fdf26dc36ffe87cf0765bd` | 10,323,128 | -| `delivery` | 100 | `5a122f783d2b501f063c718b8dc9a637de573b90c9a87a7f9f4336b1ea9c9404` | 2,536,391 | -| `instore` | 100 | `f92b9313e5476499d51b73a929cc901b7d87cd7c3315df56942d62c83aa34407` | 5,290,647 | -| `ota` | 100 | `874e9117f94758a33ec565ab43ed53361bf09253e157b6f3174e0e5dbf32cbde` | 8,732,896 | +| `cross_domain` Chinese | 100 | `a7e782785f40667511a54562d613e2122de645a36b8661a5658c1740d34a5815` | 10,318,151 | +| `cross_domain` English | 100 | `54a396cd2e5de2a26da29b954341ad190806ed7f2e870dd96511e81a947c2865` | 7,050,913 | +| `delivery` Chinese | 100 | `22746ae23eb6dbed2ee6a5e3a0f476af60e8a43bab0ea2d3ead54f0ef3829f94` | 2,572,829 | +| `delivery` English | 100 | `4906517e50ad578cdfd52426b76e4604bfd207db4254643bbc150891240f5500` | 2,081,689 | +| `instore` Chinese | 100 | `f92b9313e5476499d51b73a929cc901b7d87cd7c3315df56942d62c83aa34407` | 5,290,647 | +| `instore` English | 100 | `8e3ac7b53501d211a5261ef4e8ad9eb46bd304bd6a77794dea860f107aebaa95` | 3,828,442 | +| `ota` Chinese | 100 | `c9208c25accef24f4798bbd3f38403da081fdf48f3c7100dfc702aa7b379559e` | 8,732,896 | +| `ota` English | 100 | `f04ac51d84452557f4a68b6cc25f854edc88b0a501c85e48943859be7ab203b9` | 5,883,455 | The audit intentionally does not inline task instructions, user scenarios, or judge prompts. They are benchmark inputs in the canonical files above, and @@ -33,18 +41,21 @@ printing them into OpenBench documentation is unnecessary for reproducibility. ## Official run settings -Observed from `src/vita/config.py`, `src/vita/cli.py`, and `src/vita/run.py`: +Observed from the paper, `src/vita/config.py`, `src/vita/cli.py`, and +`src/vita/run.py`: - Default domain: `delivery,instore,ota` for cross-domain evaluation. - Default agent implementation: `llm_agent`. - Default user implementation: `user_simulator`. - Default agent LLM: `gpt-4.1`. -- Default user-simulator LLM: `gpt-4.1`. +- Main-results user-simulator LLM: `gpt-4.1-2025-04-14`; the repository uses + the shorter configurable name `gpt-4.1`. - Default evaluator LLM: `anthropic.claude-3.7-sonnet`. - Default evaluation type: `trajectory`. - Default maximum steps: 300. - Default maximum consecutive errors: 10. -- Default trials: 1. +- CLI default trials: 1; paper main results: 4 trials per task. +- Main-results temperature: 0.0 for all model roles. - Default seed: 300. - Default language: `chinese`. @@ -52,6 +63,11 @@ Observed from `src/vita/config.py`, `src/vita/cli.py`, and `src/vita/run.py`: costs. The official runner reads that configuration and calls external model APIs directly. +The current canonical repository is newer than the frozen +`meituan/vitabench` mirror previously audited. Its January 2026 update rectifies +tasks and tools, adds English manifests, changes run ordering, and updates +reasoning-content handling. The two revisions must not be mixed in one score. + ## Scoring protocol All public evaluation modes route through `TrajectoryEvaluator` and call @@ -71,12 +87,29 @@ The simulator also calls `generate()` for the user role. Even the registered `dummy_user` path is implemented as an LLM-backed user class, so it is not a deterministic replacement for the official user simulator. -## OpenBench compatibility finding +## OpenBench implementation path + +The upstream environment is local Python state, not a browser or external +website snapshot. Its tools use OpenAI function schemas and do not execute +model-generated code, so the next implementation slice is bounded: + +1. pin current source commit `742e240` and lock its transitive runtime; +2. adapt the target-agent calls to Inspect's evaluated model while keeping the + upstream orchestrator, task state, prompts, and tool semantics unchanged; +3. require explicit auxiliary configurations for `gpt-4.1-2025-04-14` and + `claude-3.7-sonnet`, failing closed when either is absent; +4. expose four epochs and preserve the upstream all-rubrics binary reward; +5. differential-test a small credentialed set against the pinned upstream CLI. + +VitaBench 2.0 is a separate long-term personalization benchmark and must use a +separate identity; it is not a protocol update that can silently replace 1.0. + +## Local source smoke -OpenBench can support agentic tasks when the environment and scorer are -deterministic or explicitly parameterized, as with AgentDojo and tau-bench. A -faithful VITA-Bench integration would need new support for auxiliary LLM roles -and a configured judge model in addition to the evaluated model. Without that, -any OpenBench task would either omit the official user simulator, omit the -official judge, or hard-code extra paid model dependencies. The candidate is -therefore blocked rather than approximated. +The pinned current source installed successfully into an isolated Python 3.13 +environment through `uv`, resolving 110 packages. Without model credentials, +its official loader imported every Chinese and English manifest: 100 unique +tasks in each of delivery, in-store, OTA, and cross-domain for both languages. +The first cross-domain task in each language also constructed successfully with +all 66 unique tools. This closes the source/runtime discovery blocker; +credentialed auxiliary-model parity remains the implementation gate. diff --git a/src/openbench/evals/ojbench/Dockerfile b/src/openbench/evals/ojbench/Dockerfile index 741a7ed1..3fa75dcb 100644 --- a/src/openbench/evals/ojbench/Dockerfile +++ b/src/openbench/evals/ojbench/Dockerfile @@ -1,4 +1,4 @@ -FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51c34de3f08cbb7 AS builder +FROM python:3.11-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 AS builder ARG DMOJ_REVISION=f098cd3a49a60186d1fadde5132329ec5f4f2213 ARG OJBENCH_REVISION=5e94480b1e135b98855cf5bc81213c256aff5b17 @@ -22,7 +22,7 @@ RUN python -m pip install --no-cache-dir \ && python -m pip wheel --no-build-isolation --no-cache-dir --no-deps \ --wheel-dir /wheels /src/ojbench -FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51c34de3f08cbb7 +FROM python:3.11-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 RUN apt-get update \ && apt-get install --yes --no-install-recommends g++ libseccomp2 pypy3 \ diff --git a/tests/test_ojbench.py b/tests/test_ojbench.py index ed7ccd3a..184694ea 100644 --- a/tests/test_ojbench.py +++ b/tests/test_ojbench.py @@ -146,6 +146,14 @@ def test_compose_declares_required_dmoj_boundary(): assert service["security_opt"] == ["no-new-privileges:true"] +def test_dockerfile_pins_bookworm_for_validator_compatibility(): + dockerfile = ( + Path(__file__).parents[1] / "src/openbench/evals/ojbench/Dockerfile" + ).read_text() + assert dockerfile.count("python:3.11-slim-bookworm@sha256:") == 2 + assert "python:3.11-slim@sha256:" not in dockerfile + + def test_registry_entries(): assert BENCHMARKS["ojbench_python"].function_name == "ojbench_python" assert BENCHMARKS["ojbench_cpp"].function_name == "ojbench_cpp"