feat(tasks): add LiveOIBench — olympiad C++ problems with subtask scoring

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• New benchmark: LiveOIBench — 403 informatics-olympiad problems from 72 contests across 14 olympiads (2023–2025), graded in C++ with IOI-style subtask partial credit and ranked against the contests' real human contestants. Paper: arXiv:2510.09595 (ICML 2026). Upstream toolkit: LiveOIBench-Evaluation @ 7759e3b8, Apache-2.0.
• This is the first benchmark here whose grading needs a compiler. Every other code benchmark posts to the vendored code-evaluator, which ran Python/JS/TS only. LiveOIBench is C++-first (the paper measures C++ and reports Python ~9 points lower on a USACO subset), so vendor/code-evaluator gains source="liveoibench": g++ compile, then one child per test case under RLIMIT_CPU / RLIMIT_AS at the problem's own limits with upstream's explicit 20% buffer, plus its CPU poller and its output comparison.
• Scope is 380 of 403 problems. The dataset filters task_type != "batch", so the 23 interactive problems (needing an interactor process the evaluator does not run) are excluded rather than scored zero, which would understate every model invisibly. The 5 script-judged problems run evaluate.sh shipped as data — the path sieval/tasks/CLAUDE.md forbids — and are all interactive, so the same filter covers them. These numbers are therefore not comparable to the paper's 403-problem table, which report.json's n_problems and the task's reference_impl.notes both say.
• No checker path exists, and that is fidelity. The published dataset materializes no checkers/ directory, so upstream's own judge compares outputs directly on this data and the 36 min-score subtasks (of 3700) collapse onto the all-or-nothing rule. The branch is still ported, so a release that does ship checkers is not silently mis-scored.
• Operational: the test corpus is 33.5 GB across three single-row-group parquets, so no cheap per-problem read exists. scripts/materialize_liveoibench_tests.py unpacks it once into upstream's own {competition}/{year}/{round}/{task}/tests layout (an existing $LIVEOIBENCH_ROOT/data tree is reusable via tests_root=), and the loader refuses to run until it has, naming the command. The evaluator then reads test directories off disk — inlining would move the whole corpus over HTTP once per rollout (~140 MB per problem).
• Upstream samples n=8 and reports the best candidate (not mean, not pass@k). The task defaults to n=1 and reproduces best-of-n at whatever n it is given.

Review fixes (commits 2–3)

A review of the first commit found four divergences in the human-ranking and
aggregation half — the part that produces the numbers a paper is compared
against. Two of them were documented as not existing. All are fixed here, with
counts measured against the pinned dataset revisions.

• USACO never joined the contestant table. The contest id was derived as {competition}-{year}-{round}; the published table keys USACO by division (USACO-2023-December_Contest-platinum / -combined). Of the 61 contest ids the 380 batch problems derive, 11 were absent from the 72-row table — 132 problems, 35% of the scored set, silently dropped from human_percentile / medal_rate / gold_rate behind a count-only warning. Upstream's routing is ported instead: build_problem_to_contest_map reads the authoritative mapping out of each contestant row's own problems column (the only place the split is stated), and resolve_contest_id walks upstream's fallback ladder, CCO rename included. All 380 now join.
• Recalculated_Total is in the published data. The module claimed no contest carried it; 10 of 72 do, and upstream prefers it over the per-task sum wherever it exists. The two differ for 20–83% of contestants on each, and the resulting percentile differs on four contests — at a model total of a quarter of the maximum, CEOI-2023 read 75.0 where upstream reads 37.5. Ported, dropping unparseable rows as upstream's .dropna() does rather than zeroing them, along with the Canadian Computing Olympiad rank re-derivation nested inside it.
• relative_score was problem-weighted, where upstream means inside a contest and then across contests. Measured over the 380 scored problems: 72 contests sized 1–10, the largest being JOI-2023-JOI_spring at 10, then eleven USACO rounds at 9 each — so the two are not interchangeable. This is the task's score_key.
• pass_rate meant the opposite of upstream's. What was reported as pass_rate is upstream's tests_passed_pct (fraction of test cases); what was reported as ace_rate is upstream's pass_rate (fraction of problems fully solved). Both now carry upstream's names, so report.json reads against {model}_contest.json without a translation table.

Two contests-with-no-contestants paths came out of the same pass: a contest
publishing cutoffs and an empty ranking still earns its medal (upstream's
df.empty branch — every USACO row in the release is that shape), while a
USACO -combined contest reports neither, because upstream scores it from
promotion thresholds that live in its repository rather than in the dataset. NaN
cutoff cells now read as "no cutoff" instead of as a cutoff nothing can clear.

Robustness, from the same review:

• The evaluator raised RuntimeError: coroutine raised StopIteration on a suite with no test case — it looked for a first failure among no results. It now refuses the suite by name, and the task turns an empty verdict vector into a NonRetriableSampleError naming the directory instead of scoring every subtask at zero.
• Materialization was not atomic. An interrupted 33.5 GB unpack left a partial directory that both the script's skip check and the loader's require_tests guard read as complete. Cases now land in a sibling tests.partial and are renamed over the target in one step.
• The per-request HTTP deadline was a bare float, which httpx also applies to the wait for a free connection. These deadlines span ~150 s to half an hour, so a small problem queued behind large ones could fail on the pool without being graded; pool=None decouples them.
• The test directory is read once per sample from a worker thread rather than once per rollout on the event loop.
• pyarrow is declared: imported directly by the dataset, it only reached the environment as a transitive of datasets. Re-locked with --update-reuse; the only change is the content hash.
• docker/Dockerfile.cpp removed. Dockerfile.multipl-e (which landed on main meanwhile) is byte-identical apart from its extra toolchains, and its apt-get install g++ already covers the -static link. Measured in the real base rather than assumed: on the current python:3.10-slim (Debian 13.6) g++ alone brings libc6-dev 2.41 + libstdc++-14-dev, both static archives resolve, and the judge's exact command line builds and runs a bits/stdc++.h program. The deleted file had also drifted — it pinned libstdc++-12-dev while the base moved to gcc 14.

Review fixes (commits 4–5)

Rebased onto a0ba68de (Ag-LiveCodeBench-X, #123), which touched the same two
vendored files. Both conflicts were "both added" — a new elif sample.source
arm and new optional fields on the flat response model — and both arms are kept
whole. sieval/community/liveoibench/ is untouched by this commit, so the
port's arithmetic is unchanged.

• status is now experimental, not stable. In-tree the word means one thing — faithful port, published anchor unverified or unreachable — and that is exactly this task: no model has been run; the paper's table is over 403 problems where this scores 380; and upstream publishes no model outputs (no submission_results/, no results CSV, no solutions repo), so the QuoteBench route of anchoring the grader at zero model cost is not available either. agieval is the precedent in the other direction — a full 7,272-row run, still experimental, because a clean run is not an anchor. The reasoning now ships in reference_impl.notes instead of living only here, since a PR description is not published.
• A failed sample no longer enters the human comparison. It was placed in its contest at score 0 under its real task name, which matched that problem's human column — so the model's total lost the points and the human totals gained a column for a problem that was never graded. Upstream re-totals the humans over exactly the tasks the model was scored on. Failed samples still count at zero in relative_score and pass_rate, where the denominator is every requested sample; a contest whose problems all failed now reports no percentile rather than a percentile of zero.
• An empty rollout list is refused by name. next(r for r in rollouts ...) over no rollouts raised StopIteration out of a coroutine — re-raised by Python as a bare "coroutine raised StopIteration" naming neither the sample nor the cause. The same failure shape the evaluator's empty-suite guard exists to avoid, one layer up.
• A task-name collision inside one contest is announced. Keying the model's scores by task name is what matches a human column; two problems sharing a name would collapse into one entry and silently drop the other from both the model total and the matched columns. No published contest does this (0 of 72), so it says so rather than scoring a smaller contest.
• A NULL subtasks cell stays absent instead of being coerced to "{}", which passed the loader's own broken-join check and only surfaced per sample at grading time. The "no test parquet found" guard now tracks whether a file was read rather than whether any rubric was — two different failures, the second reporting the first's message.
• LiveOIBenchDatasetSample declares the three columns it passes through but never reads (id, setup_script, evaluation_script). VENDORED.md states the trust boundary test_dir assumes (a client-named host path, so: in-cluster callers, mounted corpus, service not exposed — otherwise run with inline_tests=True), and server.py records why the two CPU fields are 0.0 on this source rather than missing.

Commit 5 then trims the prose those two review passes accumulated — 41 lines in,
65 out. The arguing came out (which upstream directories are absent, how
QuoteBench anchored instead, an agieval citation); the scope caveat, every
divergence from upstream, and the measured counts a reader would re-verify
against all stayed.

Related Issues

None.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check) — must be run without explicit paths; an explicit path overrides the sieval/community + vendor excludes and surfaces upstream's own response: any, which is kept byte-identical on purpose.
• Unit tests pass — 7417 total green, 88 of them LiveOIBench's (5 added in commit 4: the empty-rollout guard, the failed-sample exclusion, the all-failed contest, the name collision, the NULL rubric).
• Each commit-4 fix reverse-mutated at its own site — 5 mutants, 5 caught.
• python scripts/check_preflight.py — 25/25 pass, zero warnings.
• scripts/sync_meta_index.py and scripts/sync_package_stubs.py --check both clean (the PostToolUse hooks do not fire on a rebase).
• CI green on the rebased head — 9/9, both test-matrix jobs included (3.12 / 3.13). First run the branch has had: it was DIRTY before the rebase, so GitHub executed no checks at all on the previous head.

Manual

• Evaluator C++ path against real g++ 14.2, asserting the verdict vector for each shape that decides a score: all-correct, partial ([True, True, False]), TLE (killed at the buffered CPU limit), MLE, compile error (all-False, not a skipped rollout), single-number float tolerance at 1e-6, a grader-linked submission (grader.cpp + {task}.h compiled with the solution), and a suite with zero cases. Directory mode and inline mode produce identical verdicts; a .in with no .out is a loud error.
• Ranking port diffed against upstream's own pandas implementation on the real pinned contestant parquet — score_contest vs generate_rankings.compute_human_metrics across 360 (contest × score-level) pairs: zero mismatches in percentile or medal, and build_problem_to_contest_map returns an identical 403-entry map.
• Full end-to-end run with the vendored server under uvicorn: dataset (real parquet files) → upstream's prompt → extraction from a model-shaped reply → real g++ over HTTP → subtask scores → report. A fully-solvable problem scored 100/100; a problem with one unreachable subtask scored 40/100 with subtask 2 at zero; best-of-n picked the 100-point rollout over a 40-point one.
• Port fidelity re-verified against upstream at 7759e3b8: code_extractor.py is byte-identical below its provenance header (diff is 0 lines); interprete_task_result and total_points match line for line; prompts.py matches process_dataset._write_prompt exactly, guard and float rendering included.
• Dataset claims re-measured off the pinned revisions: 403 problems / 72 contests, 380 batch + 23 interactive, all 380 joining a contestant row (0 dropped), Recalculated_Total on exactly 10 of 72, 22 USACO rows with cutoffs but empty rankings, 17/403 satisfying the prompt's grader block, 41 shipping a grader bundle, all 5 script-judged problems interactive, and cc-by-4.0 on all three HF repos.
  No model has been run, so there is no score-comparison table and the task
  ships with no alignment card. That is what status="experimental" records, and
  it is the ship state this PR is asking for rather than an outstanding item.

Running it

Not merge blockers — this is what a first real run needs:

1. Build the evaluator image from vendor/code-evaluator/docker/Dockerfile.multipl-e and deploy it (the base image has no toolchain, and every submission fails identically without one).
2. sieval dataset download liveoibench (33.5 GB) then python scripts/materialize_liveoibench_tests.py.
3. Run examples/liveoibench.yaml at n=8 against a model the paper reports. Expect the 380-vs-403 problem-set difference to show up as a gap even on a faithful port — say which set was run next to any number. human_percentile covers only the contests that publish a contestant list, which excludes all of USACO; n_contests_ranked and n_contests_medalled in report.json say how many each metric was computed over.

An aligned run at n=8 is also what promotes the task to stable.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files — and none in the five commit messages either (checked over the whole range; single author Ethan <[email protected]>, one correctly-formed Co-Authored-By trailer)
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — docker/Dockerfile.cpp is the only deletion; every reference to it (VENDORED.md, examples/liveoibench.yaml) was updated in the same commit, and Dockerfile.multipl-e's toolchain comment now records its second consumer so a later edit cannot drop g++ on the grounds that MultiPL-E is the only caller.

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table — n/a for an experimental port: no model has been run, which is exactly what that status means here and matches every other unanchored port in the tree.
• Dataset loading — the three hf: sources are revision-pinned and resolve (check_datasets passes; all three repos confirmed ungated, cc-by-4.0, SHAs read off the HF API). The 33.5 GB download itself was not run; the loader was exercised end-to-end against locally written parquet files of the published schema, and the problem/contestant parquets (920 KB / 559 KB) were downloaded at their pinned revisions to verify the joins and counts quoted above.
• Task registered in package-level __init__.py (lazy export; get_task_class("liveoibench_0shot_gen") resolves, asserted in tests)

If: community/ Changes

• Upstream diff documented — every module carries a commit-pinned upstream URL and a "divergence from upstream" note. code_extractor.py is byte-identical below its provenance header. prompts.py / scoring.py / rankings.py are adaptations: upstream reads a reconstructed problem directory and a pandas DataFrame, these read the parquet fields and the JSON rows those were written from; the arithmetic and the assembled strings are unchanged, including the limits rendering as the floats the parquet carries (2048.0, not 2048). Upstream's misspelled interprete_task_result is kept so the two can be diffed. rankings.py also ports normalize_contest_identifier, build_problem_to_contest_map and the contest-routing block of group_problems_by_contest.
• License attribution preserved (Apache-2.0 upstream; per-module provenance URLs)

Vendored evaluator (vendor/code-evaluator)

Patch documented in VENDORED.md, to be landed in scitix/code-evaluator and re-vendored. Four choices worth a reviewer's attention:

• Compilation is bounded (timeout, default 60 s) where upstream's subprocess.run has none — a submission whose template expansion never terminates would otherwise wedge the judge.
• Every test always runs. The LiveCodeBench path short-circuits at the first failure; doing that here would score every partially-correct submission as zero, since a subtask is scored on its own test group.
• A suite with no test case is refused by name, not graded. It is a deployment fault (a half-written materialized directory), and both readings it otherwise gets are wrong: an empty verdict vector scores every subtask at zero, and the failure message has no first failure to name — which raised StopIteration out of a coroutine.
• case_verdicts + case_names are returned so the client maps verdicts onto subtasks by the names the server used. The alternative — the client re-deriving the directory ordering — is a second copy of a rule that agrees until one side changes.

exec_cpp.py is not the same C++ path as the cpp row in exec_lang.LANGUAGES, and the two are not merge candidates: that one is direct-run (one program, one all-or-nothing verdict), this one compiles against the problem's own grader and runs one child per official test case under that problem's limits, returning the whole verdict vector — the only shape IOI subtask scoring can be computed from. They share no code and are reached by different source values. The test model is still named LiveCodeBenchTest though two sources now share it; renaming would widen the diff against upstream without changing the wire format.

🤖 Generated with Claude Code