feat(tasks): add MultiPL-E — HumanEval + MBPP in 24 languages - #138
Merged
Conversation
ethan-scitix
added a commit
that referenced
this pull request
Sep 10, 2026
… half held Six things from review of #138, none in the grading path. * The chat prompt sent TWO CONSECUTIVE `user` messages. Every chat template that enforces strict alternation rejects that outright -- Mistral's raises "After the optional system message, conversation roles must alternate user/assistant/..." -- and nothing between the task and the server merges same-role messages (`normalize_chat_input` passes them through as written), so the split reached the API. Upstream does not have this shape either: DSPy appends a signature's docstring as a `system` message, so the instruction belongs there. One role, and it is simultaneously the portable choice and the faithful one. Roles are now asserted on, not just the joined text -- the text is identical either way, which is why the old test could not see it. * Upstream's OUTPUT field description ("The complete program including the full prefix") was dropped with the DSPy scaffolding, while the notes claimed "field descriptions" were carried. It restates the prefix-repetition requirement the blank-prompt grading path depends on, so it is carried and the claim narrowed to say which descriptions and why. * `GET /languages` answered for the SOURCE TABLE, so the setup probe only moved its own failure one step: from "language not in the table" to "row present, toolchain absent". Measured, the second reads as `failed: [FileNotFoundError]` or `exit 127` per sample -- bucketed as `n_execution_errors`, i.e. blamed on the model's program -- which is the run of zeros the probe exists to prevent, reachable by deploying any of the other Dockerfiles with this server. Rows are now gated on `toolchain_present`, and `POST /evaluations` applies the same test so the two endpoints agree and the message names the missing command. It is a PATH lookup, not an invocation: 0.072 ms for the whole table, so nothing is compiled to answer a probe. It proves the entry point exists, not that the toolchain works -- `perl` is present here while `Test::Deep` is not. * The empty report omitted six keys the full path always writes. A schema that depends on whether any sample survived reads, to anyone diffing two runs, as a measurement gone missing rather than an empty one. Zeroed, intervals still absent. Merged with `|` and NOT spread with `**` inside the literal: a `**` makes a key unnameable to `check_report_declarations`, which then stops verifying this report's `score_key` while staying PASS -- caught only because the count in its summary line moved 55 -> 53. * `_advertised_languages` left `raise_for_status` bare, so a 5xx on the capability probe surfaced as `HTTPStatusError` with a URL and no reading -- the one branch of that probe with no guidance beside 404 and a dead socket. * `_spawn` slept 0.1s for a monitor it had not started, costing every compiled language that on each grade; and `ulimit -v` is documented as not generalising to a managed runtime, since a JVM or Go row reserves virtual space far past what it commits and would fail to start under any useful cap. Memory is a third thing a new language needs decided, beside its toolchain and its test harness's dependencies. Vendor changes carry no tests under `tests/` on purpose (VENDORED.md: they belong in `scitix/code-evaluator`); the toolchain gate was verified by probe -- an injected row with an absent command is withheld from `/languages`, refused by `/evaluations` naming that command, and a present row still executes. 6786 tests pass; ruff, ty and the full preflight clean. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
15 tasks
Four tasks over two new datasets, from two independent choices: suite (`humaneval` / `mbpp`) and protocol (completion / chat). A problem is a partial program; the model continues it; the graded program is compiled and run against the suite's assertions, with the exit code as the verdict. Language selection is a dataset arg, MMMLU-style, using upstream's registry tags rather than English names (`jl`/`ml`/`rkt`/`adb`) — those tags are the HuggingFace config names, so a wrong one is an unresolvable config instead of a language that scores badly. MBPP has 23 languages, not 24: upstream translated HumanEval to Dart and MBPP not. Two upstream details a port cannot guess: - Program assembly is `prompt + completion + "\n" + tests`, separator included (`evaluation/src/main.py`). - The chat protocol grades the MODEL's copy of the prefix and discards the dataset prompt. Upstream blanks it deliberately, because models paraphrase docstrings and failing a program over that would be a spurious error. Verified the trap: prepending the dataset prompt to a whole-program reply defines the function twice and fails in all four validated languages, so a naive port scores ~0 across the benchmark. That is why the protocols are separate tasks rather than a flag. `report.json` publishes per-language `pass@1_<tag>` + `n_problems_<tag>` (upstream publishes a table and no single number, so these are the paper-comparable columns), a size-weighted pooled headline, `pass@1_macro`, `n_languages`, and the build-vs-run failure split. Both ship `experimental`: no published-score alignment run has been made. Evaluator side, in `vendor/code-evaluator` (a local patch — land in `scitix/code-evaluator` and re-vendor): - `app/exec_lang.py` adds `cpp` / `bash` / `perl` as rows in one declarative table behind a single executor, rather than a fourth, fifth and sixth copy of the write-file/spawn/decode sequence the three hand-rolled modules each carry. Those three are untouched, so no existing task's grading moves. Commands follow upstream's `eval_<lang>.py`, including perl's rule that an otherwise-passing run fails on `ERROR` in its output. - `GET /languages`, probed at task setup before any inference. Without it an unsupported language returns a clean failed verdict per sample, so the run completes, burns a full generation budget, and reports `pass@1 = 0` with no errors — indistinguishable from a model that cannot write Racket. - `docker/Dockerfile.multipl-e` installs `libtest-deep-perl`: every MultiPL-E perl test opens with `use Test::Deep`, so without it the language scores a clean 0. A language's test-harness dependencies matter as much as its compiler. - Fixes a latent 500: for an unsupported `lang` the direct-run branch never bound `timeout`, which the log line below it reads unconditionally, so the request died with `UnboundLocalError` instead of returning its own message. That is the path every language whose toolchain is not deployed takes — 20 of 24 today. Validated end-to-end on real MultiPL-E rows for cpp / js / sh / pl (the toolchains available locally): hand-written correct and wrong completions grade as pass and fail in each, under both protocols, with per-language rates and interval declarations checked against `interval_declaration_problems`. The memory cap is verified binding rather than assumed — a 1 GiB allocation is refused at `memory_limit=256` and succeeds uncapped. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`_spawn` killed the direct child only. Anything that child forked inherited this process's stdout and stderr, so the `proc.wait()` after the kill blocked on the orphan holding those pipes rather than on the budget. The verdict was still `failed: timeout`, but the call returned whenever the orphan chose to. Measured before the fix: `bash` running a program that forks a 30s sleeper returned after 30.1s against a 3s budget, and the sleeper survived the kill. After it: 3.1s, and no survivor. The two already-bounded shapes are unchanged (no-fork busy loop 3.10s; forks-with-exiting-children 0.10s, still ok). The fix is upstream MultiPL-E's own -- `safe_subprocess` spawns with `start_new_session=True` and SIGKILLs the group, commented "Without this line, test_fork_once fails". The reap is bounded by `KILL_GRACE_SECONDS` so a process wedged in uninterruptible sleep cannot re-introduce the unbounded wait the kill exists to remove. `exec_js` / `exec_ts` have the same shape and are deliberately untouched: they are pre-existing and out of this change's scope. Also documents, in `VENDORED.md`, `README.md` and the `LANGUAGES` comment, that the per-step wall clocks diverge from upstream in BOTH directions -- upstream gives every `safe_subprocess.run` call a flat 15s, the build included, while an interpreted row here gets 3s and a build gets 60s. The old comment claimed the commands make a program that passes here one that passes there, without the budget caveat. Co-Authored-By: Claude Opus 5 <[email protected]>
…e_kind The base tier walked exactly one level. `multipl_e` sits two: leaf -> per-protocol base -> shared `feedback`, so the check found no judgement record for any of its four leaves and reported them `unverified`. That is a WARN meaning "could not read this one" -- which is exactly what it says when the declaration underneath is also wrong, so the guard went quiet on the first family to nest twice while the run still ended green. Before: `[WARN] check_reference_kind -- 4 task(s) build no judgement record this check can reach` alongside `[PASS] ... all 70 task(s)`. After: `[PASS] check_reference_kind -- all 74 task(s) ...`, no WARN. Termination is a visited set keyed on (module, class name), not a depth cap: a cap is a second number to get wrong the next time a family nests deeper, which is this bug. The helper tier stays one hop -- a helper calling a helper is a refactor away from being inlined, but a class hierarchy is a shape a benchmark family picks deliberately. `_resolve_base_class` looks in the class's OWN module before an import: a shared base and the subclasses parameterizing it commonly live in one `_base.py` (`multipl_e`, `arc`), where there is no `ImportFrom` to resolve. The mirror test is reverse-mutation checked: pinning the walk back to depth 1 fails `test_a_base_chain_is_followed_past_its_first_level` and nothing else. Co-Authored-By: Claude Opus 5 <[email protected]>
…surface
Four separate things, all in the same family:
* `_failure_buckets` matched the evaluator's message by SUBSTRING. The message
tail carries the compiler's own output verbatim, so a build failing on an
identifier named `timeout` (`error: no member named 'timeout'`) landed under
`timeouts` -- the one bucket that says the model's program ran and was slow.
Now matched on the prefix, against the evaluator's documented vocabulary. A
build that exceeds its own wall is a BUILD failure: the run never started,
and build-versus-run is the split these three keys exist to carry.
* `_advertised_languages` derived the probe URL with a bare `rsplit("/", 1)`.
A `code_eval_api` with a trailing slash dropped the empty last segment
instead of `evaluations`, probing `.../evaluations/languages` -- a 404,
reported as "this deployment predates table-driven languages" when the
deployment is fine. `rstrip("/")` first.
* An EMPTY `languages` list read as "all", which is 24 languages of inference
apart from what the caller meant. An empty list reaches the loader from a
config that computed a selection and came up with nothing far more often than
from an author spelling "everything" as `[]` -- which omitting the argument
already says. Refused, naming both readings.
* `EVALUATOR_LANG_BY_TAG`'s comment claimed upstream's `containerized_eval`
accepts both the tag and the English spelling, so "neither side is being bent
to fit the other". Nothing here is passed to `containerized_eval` at all, and
its `EVALUATORS` keys are the tags. The right column is this service's
vocabulary and only ever this service's.
`_SHARED_NOTES` gains divergence (4): the per-step wall clocks are the
service's, not upstream's -- upstream gives every `safe_subprocess.run` call a
flat 15s including the build, while an interpreted row here gets 3s and a c++
build 60s. Both directions are reachable. `meta/index.json` regenerated.
Co-Authored-By: Claude Opus 5 <[email protected]>
… half held Six things from review of #138, none in the grading path. * The chat prompt sent TWO CONSECUTIVE `user` messages. Every chat template that enforces strict alternation rejects that outright -- Mistral's raises "After the optional system message, conversation roles must alternate user/assistant/..." -- and nothing between the task and the server merges same-role messages (`normalize_chat_input` passes them through as written), so the split reached the API. Upstream does not have this shape either: DSPy appends a signature's docstring as a `system` message, so the instruction belongs there. One role, and it is simultaneously the portable choice and the faithful one. Roles are now asserted on, not just the joined text -- the text is identical either way, which is why the old test could not see it. * Upstream's OUTPUT field description ("The complete program including the full prefix") was dropped with the DSPy scaffolding, while the notes claimed "field descriptions" were carried. It restates the prefix-repetition requirement the blank-prompt grading path depends on, so it is carried and the claim narrowed to say which descriptions and why. * `GET /languages` answered for the SOURCE TABLE, so the setup probe only moved its own failure one step: from "language not in the table" to "row present, toolchain absent". Measured, the second reads as `failed: [FileNotFoundError]` or `exit 127` per sample -- bucketed as `n_execution_errors`, i.e. blamed on the model's program -- which is the run of zeros the probe exists to prevent, reachable by deploying any of the other Dockerfiles with this server. Rows are now gated on `toolchain_present`, and `POST /evaluations` applies the same test so the two endpoints agree and the message names the missing command. It is a PATH lookup, not an invocation: 0.072 ms for the whole table, so nothing is compiled to answer a probe. It proves the entry point exists, not that the toolchain works -- `perl` is present here while `Test::Deep` is not. * The empty report omitted six keys the full path always writes. A schema that depends on whether any sample survived reads, to anyone diffing two runs, as a measurement gone missing rather than an empty one. Zeroed, intervals still absent. Merged with `|` and NOT spread with `**` inside the literal: a `**` makes a key unnameable to `check_report_declarations`, which then stops verifying this report's `score_key` while staying PASS -- caught only because the count in its summary line moved 55 -> 53. * `_advertised_languages` left `raise_for_status` bare, so a 5xx on the capability probe surfaced as `HTTPStatusError` with a URL and no reading -- the one branch of that probe with no guidance beside 404 and a dead socket. * `_spawn` slept 0.1s for a monitor it had not started, costing every compiled language that on each grade; and `ulimit -v` is documented as not generalising to a managed runtime, since a JVM or Go row reserves virtual space far past what it commits and would fail to start under any useful cap. Memory is a third thing a new language needs decided, beside its toolchain and its test harness's dependencies. Vendor changes carry no tests under `tests/` on purpose (VENDORED.md: they belong in `scitix/code-evaluator`); the toolchain gate was verified by probe -- an injected row with an absent command is withheld from `/languages`, refused by `/evaluations` naming that command, and a present row still executes. 6786 tests pass; ruff, ty and the full preflight clean. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Comments and docs only -- no behaviour change, and every fact kept. The previous commit explained several points twice and hedged where the measurement already spoke; -35 lines net across the seven files it touched. Cut, specifically: the second telling of why the chat instruction is a `system` message (the docstring and `reference_impl.notes` each carried the full Mistral reasoning -- the notes keep it, the docstring points at it), the restatement of "offered vs present" in three places (`/languages`, README, VENDORED), and the prose around `**` vs `|` in the empty report, which now names the consequence and stops. Kept verbatim where the number IS the argument: 0.1 ms for the PATH sweep, the build-vs-run entry-command rule, and why an address-space cap does not suit a managed runtime. `reference_impl.notes` changed, so `meta/index.json` is regenerated with it. 6786 tests pass; ruff, ty and preflight clean, with `check_report_declarations` still resolving 55 score_keys (the count the `**`-spread regression moved). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
ethan-scitix
force-pushed
the
feat/multipl-e
branch
from
September 10, 2026 16:02
6593304 to
2926ac0
Compare
ethan-scitix
added a commit
that referenced
this pull request
Sep 10, 2026
Review follow-up. The prefix set was missing one spelling that the substring test it replaced had caught by accident, so the fix came out stricter than intended in exactly one place. `failed: [CaseTimeout]` is the per-case wall arriving late. `_unsafe_execute` catches `CaseTimeout` per case, but a signal the kernel has already delivered cannot be un-delivered: an alarm firing as a case ends is raised at the next bytecode check, which can land past that `except`. The worker's outer handler then formats it through the generic class-name branch, so a real wall reads `failed: [CaseTimeout] ` instead of `failed: case timeout: 6.0s`. The vendored service names `CaseTimeout` in that handler for precisely this reason. Reachable only where a per-case budget is armed, which is the two LiveCodeBench tasks (both default `timeout_per_case=6.0`); HumanEval, MBPP and SciCode append the test to the program and take the `exec_py_code` path. Still diagnostic only. The name cannot come from submitted code -- it is a service-internal `BaseException` -- so the added prefix carries no false-positive risk. Also from review: - `exec_lang` and `failed: build timeout` are marked forward-looking. Neither is in the vendored service at this commit; both arrive with #138, and the js/ts entry two lines up already carried that caveat. - scicode's comment claimed its `TimeoutError` test was "matched the way `memoryerror` below is". It is not -- the new one anchors on the brackets that `failed: [{type}] ...` always supplies, while the neighbours match bare. The comment now says so. The neighbours are deliberately left alone: bracketing them would narrow `memory_errors` and `import_errors` without a measurement, which is the same scope line this change already draws. - The convention survey's `zip(..., strict=False)` now says why it is not strict: `n` ops against `n + 1` operands, so `strict=True` would raise on every chained compare in the tree. Prose trimmed throughout. The three-line comment repeated verbatim at five identical call sites is gone -- the helper's name and its module carry it -- and the module and test docstrings say the same things in fewer words. Reverse-mutation: dropping the new prefix fails two tests, one of them the pairwise case asserting that `[CaseTimeout]` and `[TimeoutError]` split opposite ways. 6798 tests pass; ruff, ty, both sync checks and preflight clean. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
ethan-scitix
added a commit
that referenced
this pull request
Sep 10, 2026
Six tasks decided `timeouts` with `"timeout" in msg.lower()`. The code-eval
service documents a small closed set of message shapes, but every one of them
interpolates the failing program's own output after the colon -- so the
substring test also charged these to the clock:
failed: [TimeoutError] deadline exceeded the program raised
failed: [ValueError] timeout must be positive
failed: output ['timeout'] != expect ['ok'] LiveCodeBench, wrong answer
failed [build exit 1]: error: no member named 'timeout' in 'Config'
`timeouts` is the one bucket that says the program RAN and was merely slow, so a
misfiled failure sends a reader after efficiency instead of the exception or the
wrong answer in front of them.
Diagnostic only, and that is why it survived six copies: `correct` comes from
the service's boolean, never from this string, so no headline moves and no
metric test could have failed.
The vocabulary now has one home (`_code_eval_msg.py`) because it is one contract
with six readers and it belongs to the service, not to any benchmark -- the same
reason `_sqlite_exec.py` sits there. It is the union across executors, since the
spellings differ by which one answered: `failed: subprocess timeout:`
(`exec_py_code`, `exec_py_test`), `failed: case timeout:` / `compile timeout:`
(LiveCodeBench's per-case budgets), `failed: timeout` (js/ts), `failed: build
timeout` (compiled rows). Each is emitted only by a timeout path, so the union
has no false positives.
`scicode` keeps BOTH readings, deliberately: its `timeouts` sits beside
`memory_errors` and `import_errors`, which read exception CLASS NAMES out of the
message tail on purpose, so a raised `TimeoutError` is in scope there. Only the
word-appearing-anywhere-else case is removed. Narrowing that counter to the wall
alone would redefine someone else's metric, which a bug fix does not get to do.
Two existing fixtures asserted `timeouts == 1` from `"timeout"` and `"Timeout
exceeded"` -- strings `exec_py_code` cannot emit. They would have passed against
any implementation; both now carry the service's real wording, plus a case
proving a raised `TimeoutError` is not counted.
Re-drift is what the survey is for (`test_timeout_bucketing_convention.py`),
built like `test_grading_call_site_convention.py`: it reads the AST rather than a
hand-kept list, forbids the BARE literal only, and counts CALL nodes rather than
occurrences of the name -- an unused import satisfied the first version of that
assertion, which reverse-mutation caught. All three mutants (predicate reverted,
one call site reverted, every reader removed) now fail the suite.
Follow-up to #138, which fixed this in its own `_failure_buckets` and where the
prefixes were first enumerated. When that lands, its bucketing should adopt this
module; the branches touch no common file, so they merge in either order.
6796 tests pass; ruff, ty, both sync checks and all 24 preflight checks clean.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
ethan-scitix
added a commit
that referenced
this pull request
Sep 10, 2026
Review follow-up. The prefix set was missing one spelling that the substring test it replaced had caught by accident, so the fix came out stricter than intended in exactly one place. `failed: [CaseTimeout]` is the per-case wall arriving late. `_unsafe_execute` catches `CaseTimeout` per case, but a signal the kernel has already delivered cannot be un-delivered: an alarm firing as a case ends is raised at the next bytecode check, which can land past that `except`. The worker's outer handler then formats it through the generic class-name branch, so a real wall reads `failed: [CaseTimeout] ` instead of `failed: case timeout: 6.0s`. The vendored service names `CaseTimeout` in that handler for precisely this reason. Reachable only where a per-case budget is armed, which is the two LiveCodeBench tasks (both default `timeout_per_case=6.0`); HumanEval, MBPP and SciCode append the test to the program and take the `exec_py_code` path. Still diagnostic only. The name cannot come from submitted code -- it is a service-internal `BaseException` -- so the added prefix carries no false-positive risk. Also from review: - `exec_lang` and `failed: build timeout` are marked forward-looking. Neither is in the vendored service at this commit; both arrive with #138, and the js/ts entry two lines up already carried that caveat. - scicode's comment claimed its `TimeoutError` test was "matched the way `memoryerror` below is". It is not -- the new one anchors on the brackets that `failed: [{type}] ...` always supplies, while the neighbours match bare. The comment now says so. The neighbours are deliberately left alone: bracketing them would narrow `memory_errors` and `import_errors` without a measurement, which is the same scope line this change already draws. - The convention survey's `zip(..., strict=False)` now says why it is not strict: `n` ops against `n + 1` operands, so `strict=True` would raise on every chained compare in the tree. Prose trimmed throughout. The three-line comment repeated verbatim at five identical call sites is gone -- the helper's name and its module carry it -- and the module and test docstrings say the same things in fewer words. Reverse-mutation: dropping the new prefix fails two tests, one of them the pairwise case asserting that `[CaseTimeout]` and `[TimeoutError]` split opposite ways. 6798 tests pass; ruff, ty, both sync checks and preflight clean. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
ethan-scitix
added a commit
that referenced
this pull request
Sep 10, 2026
#138 landed while this branch was open, which makes two of the module's statements false and one of them dangerous. Stale: `exec_lang` and `failed: build timeout` are in the tree now rather than forward-looking, and "no in-tree task sends a non-default `lang`" stopped being true the moment MultiPL-E arrived with 24 of them. Dangerous: MultiPL-E routes `failed: build timeout` to `n_build_errors`, NOT to `timeouts`, and argues for it -- the run never started, and build-versus-run is the split its three keys exist to carry. `is_timeout_message` answers "did the service stop the clock", which for a build wall is yes. Those are different questions, and this branch's own note that MultiPL-E "should adopt this module" reads as an instruction to collapse them. So the module now says which of the two questions it answers, and points at the counter that deliberately disagrees. No behaviour change, and no new test: MultiPL-E already asserts `n_build_errors == 2, timeouts == 0` on that exact message in tests/unit/tasks/multipl_e/test__base.py, so a naive adoption fails there rather than silently moving a bucket. Rebased onto main (#141, #138) in the same push. 7123 tests pass -- the jump from 6812 is MultiPL-E's own suite arriving, not new tests here; ruff, ty, both sync checks and preflight clean. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
24 tasks
ethan-scitix
added a commit
that referenced
this pull request
Sep 11, 2026
…142) * fix(tasks): read a timeout off the message prefix, not by substring Six tasks decided `timeouts` with `"timeout" in msg.lower()`. The code-eval service documents a small closed set of message shapes, but every one of them interpolates the failing program's own output after the colon -- so the substring test also charged these to the clock: failed: [TimeoutError] deadline exceeded the program raised failed: [ValueError] timeout must be positive failed: output ['timeout'] != expect ['ok'] LiveCodeBench, wrong answer failed [build exit 1]: error: no member named 'timeout' in 'Config' `timeouts` is the one bucket that says the program RAN and was merely slow, so a misfiled failure sends a reader after efficiency instead of the exception or the wrong answer in front of them. Diagnostic only, and that is why it survived six copies: `correct` comes from the service's boolean, never from this string, so no headline moves and no metric test could have failed. The vocabulary now has one home (`_code_eval_msg.py`) because it is one contract with six readers and it belongs to the service, not to any benchmark -- the same reason `_sqlite_exec.py` sits there. It is the union across executors, since the spellings differ by which one answered: `failed: subprocess timeout:` (`exec_py_code`, `exec_py_test`), `failed: case timeout:` / `compile timeout:` (LiveCodeBench's per-case budgets), `failed: timeout` (js/ts), `failed: build timeout` (compiled rows). Each is emitted only by a timeout path, so the union has no false positives. `scicode` keeps BOTH readings, deliberately: its `timeouts` sits beside `memory_errors` and `import_errors`, which read exception CLASS NAMES out of the message tail on purpose, so a raised `TimeoutError` is in scope there. Only the word-appearing-anywhere-else case is removed. Narrowing that counter to the wall alone would redefine someone else's metric, which a bug fix does not get to do. Two existing fixtures asserted `timeouts == 1` from `"timeout"` and `"Timeout exceeded"` -- strings `exec_py_code` cannot emit. They would have passed against any implementation; both now carry the service's real wording, plus a case proving a raised `TimeoutError` is not counted. Re-drift is what the survey is for (`test_timeout_bucketing_convention.py`), built like `test_grading_call_site_convention.py`: it reads the AST rather than a hand-kept list, forbids the BARE literal only, and counts CALL nodes rather than occurrences of the name -- an unused import satisfied the first version of that assertion, which reverse-mutation caught. All three mutants (predicate reverted, one call site reverted, every reader removed) now fail the suite. Follow-up to #138, which fixed this in its own `_failure_buckets` and where the prefixes were first enumerated. When that lands, its bucketing should adopt this module; the branches touch no common file, so they merge in either order. 6796 tests pass; ruff, ty, both sync checks and all 24 preflight checks clean. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * fix(tasks): count a late CaseTimeout, and trim the bucketing prose Review follow-up. The prefix set was missing one spelling that the substring test it replaced had caught by accident, so the fix came out stricter than intended in exactly one place. `failed: [CaseTimeout]` is the per-case wall arriving late. `_unsafe_execute` catches `CaseTimeout` per case, but a signal the kernel has already delivered cannot be un-delivered: an alarm firing as a case ends is raised at the next bytecode check, which can land past that `except`. The worker's outer handler then formats it through the generic class-name branch, so a real wall reads `failed: [CaseTimeout] ` instead of `failed: case timeout: 6.0s`. The vendored service names `CaseTimeout` in that handler for precisely this reason. Reachable only where a per-case budget is armed, which is the two LiveCodeBench tasks (both default `timeout_per_case=6.0`); HumanEval, MBPP and SciCode append the test to the program and take the `exec_py_code` path. Still diagnostic only. The name cannot come from submitted code -- it is a service-internal `BaseException` -- so the added prefix carries no false-positive risk. Also from review: - `exec_lang` and `failed: build timeout` are marked forward-looking. Neither is in the vendored service at this commit; both arrive with #138, and the js/ts entry two lines up already carried that caveat. - scicode's comment claimed its `TimeoutError` test was "matched the way `memoryerror` below is". It is not -- the new one anchors on the brackets that `failed: [{type}] ...` always supplies, while the neighbours match bare. The comment now says so. The neighbours are deliberately left alone: bracketing them would narrow `memory_errors` and `import_errors` without a measurement, which is the same scope line this change already draws. - The convention survey's `zip(..., strict=False)` now says why it is not strict: `n` ops against `n + 1` operands, so `strict=True` would raise on every chained compare in the tree. Prose trimmed throughout. The three-line comment repeated verbatim at five identical call sites is gone -- the helper's name and its module carry it -- and the module and test docstrings say the same things in fewer words. Reverse-mutation: dropping the new prefix fails two tests, one of them the pairwise case asserting that `[CaseTimeout]` and `[TimeoutError]` split opposite ways. 6798 tests pass; ruff, ty, both sync checks and preflight clean. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * fix(tasks): read scicode's exception counters off the class slot Follow-up to the review thread on `memory_errors` / `import_errors`, left matching a bare `"memoryerror" in msg` beside the timeout fix. Same defect as the timeout one: the tail of `failed: [{type}] {e}` is the program's own output, so `[ValueError] simulated memoryerror path` counted as an OOM. Bracketing them was the obvious fix and is wrong. The service reports the CONCRETE class, so a subclass is what actually arrives, and a test for `[memoryerror]` drops it. Surveyed every exception class reachable from the SciCode execution image's stack (docker/Dockerfile.scicode -- numpy 1.26.4, scipy, sympy, matplotlib; no torch, no pyarrow): memoryerror builtins.MemoryError, numpy's (below) no loss modulenotfounderror builtins.ModuleNotFoundError no loss importerror builtins.ImportError, zipimport.ZipImportError `ZipImportError` is an `ImportError` that a bracketed test would silently stop counting. numpy is safe for a non-obvious reason worth recording: its `_ArrayMemoryError` carries `@_display_as_base`, which assigns `cls.__name__ = cls.__base__.__name__`, so the allocation failure most likely in a numpy-heavy benchmark reports as a plain `MemoryError` -- checked at the pinned 1.26.4, not only at the version installed locally. So the slot is parsed rather than pattern-matched. `exception_class_name` returns the class the service named, and each counter matches its family with `endswith`. That drops false positives only: it never stops counting a class the service actually named, which is what separates this from the metric change I said would need its own measurement first. `timeouts` now reads the same way, which makes the parity its comment used to claim untruthfully true. 14 new tests. Reverse-mutation, each direction caught by the test written for it: reverting to the bare substring fails the tail case, narrowing the slot to the builtin name fails the subclass case. 6812 tests pass; ruff, ty, both sync checks and preflight clean. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * docs(tasks): a clock stop is not the `timeouts` bucket #138 landed while this branch was open, which makes two of the module's statements false and one of them dangerous. Stale: `exec_lang` and `failed: build timeout` are in the tree now rather than forward-looking, and "no in-tree task sends a non-default `lang`" stopped being true the moment MultiPL-E arrived with 24 of them. Dangerous: MultiPL-E routes `failed: build timeout` to `n_build_errors`, NOT to `timeouts`, and argues for it -- the run never started, and build-versus-run is the split its three keys exist to carry. `is_timeout_message` answers "did the service stop the clock", which for a build wall is yes. Those are different questions, and this branch's own note that MultiPL-E "should adopt this module" reads as an instruction to collapse them. So the module now says which of the two questions it answers, and points at the counter that deliberately disagrees. No behaviour change, and no new test: MultiPL-E already asserts `n_build_errors == 2, timeouts == 0` on that exact message in tests/unit/tasks/multipl_e/test__base.py, so a naive adoption fails there rather than silently moving a bucket. Rebased onto main (#141, #138) in the same push. 7123 tests pass -- the jump from 6812 is MultiPL-E's own suite arriving, not new tests here; ruff, ty, both sync checks and preflight clean. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * fix(tasks): correct the CaseTimeout claim, and share the vocabulary Review follow-up, five findings. `CaseTimeout` is forgeable. The module claimed submitted code could not produce the name because the service's class is a `BaseException`. That is why the service's own handler names it explicitly, but it says nothing about what a program may call its own class: `class CaseTimeout(Exception)` raised from a solution is caught by `_unsafe_execute_fn_call`'s `except Exception` and wrapped into `failed: [CaseTimeout] ...`, which `is_timeout_message` accepts. Checked against the service's own helper rather than argued. The prefix stays -- dropping it loses a real wall, the substring test it replaced counted the forgery too, and what the forgery costs is one diagnostic count -- but the note now says what is true. The vocabulary is split into `CODE_EVAL_BUILD_TIMEOUT_PREFIXES` and `CODE_EVAL_RUN_TIMEOUT_PREFIXES`, union unchanged, so MultiPL-E reads the two groups instead of keeping its own copy of two of them. The fused tuple was what blocked it: its three keys split build from run, which `is_timeout_message` deliberately does not, so it could take the predicate only by moving its build walls. No behaviour moves -- none of its 24 languages is Python, so every prefix it gains is unreachable there. Routing them anyway is what keeps one home for the contract, and means a Python row added later lands in `timeouts` rather than silently in `n_execution_errors`. Also: - The prefix tuples are lowercased, and only `[casetimeout]` depends on it, so a caller comparing a raw message passes against the other five and silently drops that one. Documented, and asserted. - scicode's `import_errors` matched `ModuleNotFoundError` by equality where its two siblings match a family. The stated rule is the family, since the service reports the concrete class, so a subclass is what arrives. - The convention test justified its narrowness with scicode's `"[timeouterror]" in msg`, which 6d72735 had already replaced with a class-slot read. The carve-out is kept for the next such counter; the example is now the live one. Reverse-mutation, each caught by a test: the two groups made to overlap, a prefix left uppercased, MultiPL-E routing the run group to `n_build_errors`, and scicode's import family narrowed back to the builtin. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
Summary
humaneval24 languages /mbpp23 — upstream translated HumanEval to Dart and MBPP not) × protocol (completion / chat). Fills thecodegapdocs/benchmark-coverage/coverage.yaml:268records as reported by Kimi K2.6 and MiMo-V2.5-Pro.languages), MMMLU-style, using upstream's registry tags, not English names (jl/ml/rkt/adb) — those tags are the HF config names, so a wrong one is an unresolvable config rather than a language that quietly scores badly.vendor/code-evaluatorgains a declarative language table (cpp/bash/perlas{ext, build argv, run argv, budgets}rows behind one executor) plusGET /languages, probed at task setup before any inference. Also fixes a latent 500 on that exact path.status="experimental": no published-score alignment run has been made (see Manual below).Related Issues
None. Refs the
codecoverage gap indocs/benchmark-coverage/coverage.yaml.Test Plan
Automated
ruff check && ruff format --check)ty check)tests/unit/tasks/multipl_e/,tests/unit/datasets/test__multipl_e.py)scripts/check_preflight.py— all checks pass, includingcheck_meta_index_sync,check_report_declarations,check_reference_kind,check_datasets(source pinned tonuprl/MultiPL-E@28441b60)Manual
Executed end-to-end against a live code-eval service on real MultiPL-E rows, for the four languages whose toolchains were available (
cpp,js,sh,pl). MultiPL-E ships no reference solutions, so correct and wrong completions were hand-written per language.pass@150.0 per language at one-correct-of-two rollouts, pooled 50.0, macro 50.0.len(prompt)split and the blank-prompt assembly.build exit 1for cpp,exit 1js,exit 2sh,exit 255pl).exit 1/ERROR-on-stdout for perl; js unchanged; unsupportedlangreturns its own message instead of a 500.memory_limit=256, allowed uncapped, trivial program unaffected);cppinfinite loop andbash sleep 30both stopped by the wall.interval_declaration_problems()(what the runner applies at report-write time) returns clean at n=1/2/4, on uniform verdicts, with a pipeline fail present, and on the empty path.No score comparison table, deliberately. Producing one needs a model run at upstream's protocol (20 completions at temperature 0.2), which this change does not include — hence
experimentalrather thanstable, and theUNMEASURED:clause in bothreference_impl.notes. The per-languagepass@1_<tag>keys are the paper-comparable ones; the headline is a size-weighted pool over whatever languages ran, withpass@1_macrobeside it, because upstream publishes a table and no single number.sieval dataset download multipl_e_humanevalwas not exercised; the loader was driven directly against the Hub (641 rows over 4 languages, per-language counts 161/161/161/158 — they differ by language, which is why per-language denominators are reported).Checklist
Required (all PRs)
AI-Generated Code - Claude Opus 5 (1M context) (Anthropic)in module docstringcore/—core/untouchedexec_*modules are untouched, so no existing task's grading movesIf: New or Modified Benchmark
experimental. See Manual.sieval dataset download— loader driven directly against the Hub instead; thehf:source is pinned andcheck_datasetspasses.sieval/tasks/multipl_e/with an empty__init__.py; all four resolve by name throughget_task_class()(asserted intest_multipl_e_family.py, since a nested task that imports fine but is unresolvable shows up only as a bareKeyErrorfromsieval task show)If: community/ Changes
sieval/community/is untouched. The equivalent applies to thevendor/code-evaluatorpatch:VENDORED.mdentries covering the language table, the endpoint, the perl output rule, the reduced failure taxonomy, theulimit-over-preexec_fnchoice, and the 500 fixgh api .../licensereportsNOASSERTION, not a missing license), which is more restrictive than this repo's Apache-2.0. Its per-language graders are ~30 lines each of build command, run command and status classification, so the commands were reimplemented in the table with upstream cited per row. The dataset is plain MIT.If: Breaking Change
Not a breaking change. Purely additive: 4 tasks, 2 datasets, 3 new evaluator languages, 1 new endpoint.
CODE_EXECUTOR_MAPmoved from per-request construction to module scope with its original three rows intact.If: New Dependency
None. No
pyproject.tomlchange.Reviewer note — three PRs now add multi-language execution to the same file
Worth reconciling rather than merging blind.
vendor/code-evaluator/app/server.pyis touched by three open branches, each with a different mechanism:exec_cpp.py, ownsource="liveoibench"branch, compile-once-run-many over official case filesexec_agnostics.py,podman run ghcr.io/nuprl/agnostics@<digest>exec_lang.py, extendsCODE_EXECUTOR_MAPfor the direct-run sourcesNo semantic clash: #121's
cpplives under its ownsourceand never entersCODE_EXECUTOR_MAP, and #123 adds its ownsourcetoo. The conflicts are textual — the imports, theCODE_EXECUTOR_MAPregion, andVENDORED.md— so whoever merges second rebases.The design question is worth asking now rather than at 20 languages: for MultiPL-E's remaining languages, delegating to a container may beat apt-installing toolchains. Upstream MultiPL-E publishes its own
evaluation/Dockerfile, and #123 has already built the container-delegation plumbing (digest pinning, command-as-deployment-config,langpattern validation). This PR's table is the right shape for the four languages it proves, and each further language is currently a row plus an apt package — but if #123 lands first, the cheaper path to the other 20 is probably its mechanism with a MultiPL-E image, not 20 more apt lines. Happy to rework in that direction.One thing that generalises out of this and is worth applying to both siblings: a language's test-harness dependencies matter as much as its compiler. Every MultiPL-E perl test opens with
use Test::Deep, so withoutlibtest-deep-perlthe language scores a clean 0 with no error anywhere — upstream's own image installs it, and its Dockerfile also pinslua-unitand ajavatuplesjar for lua and java. Adding a language means reading its test template, not just installing its toolchain.🤖 Generated with Claude Code