fix(tasks): read a timeout off the message prefix, not by substring

[ethan-scitix]

Type

• fix — bug fix or alignment correction

Summary

• Six tasks decided their timeouts count with "timeout" in msg.lower(). The code-eval service documents a small closed set of message shapes, but each one interpolates the failing program's own output after the colon — so the substring test also charged these to the clock: failed: [TimeoutError] deadline exceeded (the program raised), failed: [ValueError] timeout must be positive, failed: output ['timeout'] != expect ['ok'] (LiveCodeBench, a wrong answer), failed [build exit 1]: error: no member named 'timeout' in 'Config' (the compiled-language executor from feat(tasks): add MultiPL-E — HumanEval + MBPP in 24 languages #138).
• timeouts is the one bucket that says the program ran and was merely slow, so a misfiled failure sends a reader after efficiency instead of the exception or wrong answer in front of them.
• Diagnostic only, and that is why it survived six copies: correct comes from the service's boolean, never from this string, so no headline metric moves and no metric test could have failed. Nothing any of the six scored was wrong.
• The vocabulary now has one home, sieval/tasks/_code_eval_msg.py — one contract, six readers, owned by the service rather than any benchmark (the same reason _sqlite_exec.py sits there). It is the union across executors, since the spelling depends on which one answered: failed: subprocess timeout: (exec_py_code, exec_py_test), failed: case timeout: / compile timeout: (LiveCodeBench per-case budgets), failed: [CaseTimeout] (below), failed: timeout (js/ts), failed: build timeout (exec_lang). The last two reach MultiPL-E rather than this module's readers, which are all Python. Each is emitted only by a timeout path, so the union has no false positives.
• "The service stopped the clock" is not "belongs in timeouts." feat(tasks): add MultiPL-E — HumanEval + MBPP in 24 languages #138 merged while this branch was open, which made that distinction load-bearing rather than pedantic: MultiPL-E routes failed: build timeout to n_build_errors, not to timeouts, and argues for it — the run never started, and build-versus-run is the split its three keys exist to carry. is_timeout_message answers the first question, so for a build wall it says yes. The module now states which question it answers and points at the counter that deliberately disagrees. No behaviour change, and no new test: tests/unit/tasks/multipl_e/test__base.py already asserts n_build_errors == 2, timeouts == 0 on that exact message, so a naive adoption fails there instead of silently moving a bucket.
• A prefix set also has to be complete, and the first one was not. failed: [CaseTimeout] is the per-case wall arriving late: _unsafe_execute catches CaseTimeout per case, but a signal the kernel has already delivered cannot be un-delivered, so an alarm firing as a case ends can land past that except and is then formatted by the worker's outer handler through the generic class-name branch. The vendored service names CaseTimeout in that handler for exactly this reason. The substring test caught it by accident and the first prefix set dropped it — the one place this change was stricter than what it replaced. Reachable only where a per-case budget is armed, which is the two LiveCodeBench tasks (both default timeout_per_case=6.0); the other four append the test to the program and take the exec_py_code path. Service-internal BaseException, so submitted code cannot forge the name and the added prefix carries no false-positive risk.
• scicode keeps both readings, deliberately. Its timeouts sits beside memory_errors and import_errors, which read exception class names on purpose, so a raised TimeoutError is in scope there; only the word-appearing-anywhere-else case is removed. Narrowing the counter to the wall alone would redefine someone else's metric, which a bug fix does not get to do.
• All three of those counters now read the exception class slot, not the message text. memory_errors and import_errors carried the same defect ([ValueError] simulated memoryerror path counted as an OOM), so they are fixed with it. Bracketing them would have been wrong: the service reports the concrete class, so a subclass is what arrives, and "[memoryerror]" in msg silently stops counting it. Surveyed every exception class reachable from the execution image's stack (docker/Dockerfile.scicode — numpy 1.26.4, scipy, sympy, matplotlib; no torch, no pyarrow) and zipimport.ZipImportError is a live loss under bracketing. So exception_class_name() parses the slot and each counter matches its family with endswith — which drops false positives only, and never stops counting a class the service actually named. That is what keeps it a bug fix rather than a metric change owing its own measurement.
  • numpy is safe for a non-obvious reason worth recording: _ArrayMemoryError carries @_display_as_base, which assigns cls.__name__ = cls.__base__.__name__, so the allocation failure most likely in a numpy-heavy benchmark reports as a plain MemoryError. Checked at the pinned 1.26.4, not only at the locally installed version.

Related Issues

Follow-up to #138 (now merged), which fixed this in its own _failure_buckets and is where the prefixes were first enumerated. This branch is rebased onto it. If its bucketing later adopts this module, it should take the vocabulary (CODE_EVAL_TIMEOUT_PREFIXES) and keep its own build-versus-run routing — swapping in is_timeout_message wholesale would move build walls into timeouts, which is the one thing that split exists to prevent.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check)
• Unit tests pass — 7123 passed, of which 36 are new (counted by collection; the jump from 6812 is MultiPL-E's own suite arriving with the rebase, not new tests here)
• scripts/check_preflight.py — no failures (check_links skipped at the default level); sync_meta_index.py --check and sync_package_stubs.py --check both clean

Manual

No model run: this changes a diagnostic counter, not a score. Verified by construction instead.

• Every message in the tests is the service's own wording, taken verbatim from vendor/code-evaluator/app/exec_*.py — six timeout spellings and six near-misses. Each near-miss contains the word, so the whole file would pass trivially against a substring test; rejecting them is what the assertions are for.
• The two class-name shapes are pinned against each other. failed: [CaseTimeout] and failed: [TimeoutError] ... differ only by which class and must split opposite ways — the service's own wall versus the program's own exception. They are asserted as a pair, because getting that backwards in either direction is the whole defect: one way misfiles a wrong answer as slow, the other loses a real wall.
• Two existing fixtures were asserting on strings the service cannot emit — "timeout" and "Timeout exceeded", where exec_py_code says failed: subprocess timeout: 3.0s. They would have passed against any implementation. Both now carry the real wording, plus a new case proving a raised TimeoutError is not counted.
• Re-drift survey (test_timeout_bucketing_convention.py), built like test_grading_call_site_convention.py: reads the AST rather than a hand-kept list, forbids the bare literal only (so scicode's deliberate "[timeouterror]" in msg stays legal), and resolves the task tree from the imported package so a worktree cannot survey the primary checkout.
• The exception-counter survey is empirical, not argued. Enumerated every reachable exception class in the image's stack and checked which ones bare and bracketed matching disagree on — that is what rejected bracketing and produced the endswith design. The two failure modes are pinned by a test each, in opposite directions.
• Discriminating power proven by reverse-mutation, all six caught: predicate reverted to substring (7 failures), one call site reverted (2), every reader removed (1), [CaseTimeout] prefix dropped (2), memory_errors reverted to the bare substring (1, the tail case), memory_errors narrowed to the builtin name (1, the subclass case). The third was not caught by the first version of the reader assertion — it grepped for the name, and a leftover import line satisfied it — so it now counts ast.Call nodes.

Checklist

Required (all PRs)

• PR title follows conventional format
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - Claude Opus 5 (1M context) (Anthropic) in module docstring
• No new upper-layer dependencies added to core/ — core/ untouched
• Deleted code verified — no call site removed; six predicates rewritten in place. The only deletions are comments: the three-line note that was repeated verbatim at five identical call sites, now carried by the helper's name and the module it lives in.

If: New or Modified Benchmark

Not a benchmark change. No task's score, pass@1 or denominator_policy moves — only timeouts, plus scicode's memory_errors / import_errors, and only where each was previously wrong.

If: Breaking Change

Not breaking. timeouts will read lower on any future run whose failures happened to quote the word, which is the fix rather than a regression; historical report.json files are untouched. The [CaseTimeout] prefix moves it the other way, on the two LiveCodeBench tasks only, and only for a wall that was always a wall.

If: New Dependency

None.