feat(tasks): add QuoteBench — execution-graded shell quoting across two command paths - #140
Merged
Merged
Conversation
ethan-scitix
force-pushed
the
feat/quotebench-port
branch
from
September 11, 2026 01:50
b72fa67 to
3b57bc8
Compare
Vendors upstream's `core`, `scenarios` and `shellesc` byte-identically from LeonardNJU/quoteBench @ 693325a6 (Apache-2.0). Only the halves sieval needs: contract prompts, the Task type, the literal-text markers, and the 56 frozen tasks. `harness.py` is deliberately left out — execution is the code-evaluator's half, and vendoring it here would put an executor one import away from a task. The identity test pins each file's sha256 taken from the upstream tree rather than from the copy, because a digest read off the copy pins whatever was copied. Proven to bite by reverse mutation: substituting an ASCII lookalike for the U+27EA marker turns 2 of the 7 tests red, and restoring returns all 7 to green. One assertion was written wrong and corrected against the measurement, not the other way round: 48 of 56 instructions carry marked literal text, and the 8 that do not are exactly the `find-glob` and `bulk-rename` families. That is structural — those two put their hazards in the fixture, as hostile filenames already on disk, instead of handing the model a literal to reproduce. The test pins the split rather than the bare count so it still bites if a family that should carry markers loses them. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
QuoteBench ships no downloadable prompt set — upstream constructs all 56 tasks in Python. `scripts/gen_quotebench_snapshot.py` renders the model-facing half into `<data-dir>/quotebench/quotebench-core.json`, following the `local:` bring-your-own convention that `datasets/downloaders/local.py` defines: the file is produced by a script straight into the data dir, never committed, and an absent snapshot raises instead of yielding an empty split. The oracle, the naive probe and the fixture builders are deliberately not serialized: they are the evaluator's half, and shipping the oracle beside the prompt would put the answer in the dataset. A test asserts the row keys are exactly the five prompt fields. `local:` sources cannot carry a declared checksum — `check_datasets` accepts checksum keys only for `url:` basenames — so the pin between the snapshot and the vendored package is a test that compares staged rows against `all_tasks()`. It builds the fixture by calling the generator's own `build_rows()` rather than reimplementing it, so it cannot keep passing after the generator breaks. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A source whose unit of work is a task id plus one command, not a program plus test cases: the task builds its own filesystem fixture in Python, the reply runs inside it as a single `bash -c` payload, and the verdict is the exact final state — file bytes, argv, JSON, directory contents, or Git history. No reference command string is compared, which is why none of the existing exec_* modules could carry it. sieval itself still executes nothing. Verified through the HTTP path with TestClient, so pydantic validation and the declared response model are both in play — the latter matters because `ResourceMetrics` had to gain four optional fields rather than a per-source subclass, whose extras FastAPI would silently strip: - all 56 oracles pass; - replaying the stored replies of upstream's raw-vs-nested arm reproduces the GNU verdicts upstream recorded for them 224/224 on `passed` and 224/224 on failure class, across all four crossover cells; - protocol errors (unknown task, unknown contract, missing kwargs) answer with `data=None` while a merely wrong command answers with `data` present, so a caller can tell them apart. The contract-to-transport mapping lives here, not in the vendored package: upstream's `public_cli.command_for_transport` raises ValueError on `nested`, the spelling its own released dataset uses, so its public scorer cannot read its own release. We accept the released spellings and reject the CLI-only ones. Upstream's `get_task` rebuilds all 56 tasks per lookup, which under --workers N would be paid per request; an lru_cached index replaces it. The Task objects are reusable — setup/check close over the task's own constants and run_attempt allocates a fresh temp dir per attempt. Dockerfile.quotebench pins upstream's base digest and its seven GNU packages because QuoteBench scores BSD and GNU separately and the published crossover table is the GNU replay. The image is unbuilt and unrun so far — no container runtime here — so this ships behind QUOTEBENCH_EXECUTOR=local for now. VENDORED.md now separates upstream-bound patches from ones sieval owns in-tree by decision; this is the first of the latter. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two registered tasks over one benchmark, because a contract is not a prompt variant: it declares what happens to the reply between the model and the shell, and upstream's finding is that two matched command paths can score alike while reversing the order of 5 of 26 model pairs. One number for "QuoteBench accuracy" would hide which command path produced it, and the name is the only place that distinction can live. Which prompt backs the published nested column was measured rather than assumed: all 56 stored system messages in upstream's released raw-vs-nested arm match SYSTEM_PROMPT_NESTED_SHELL_V2 by identity, so the v1 constant the same module still ships — one-line-reply rule plus an escaping tutorial, which upstream marks contract-confounded — is the wrong one to carry. No extraction, deliberately: upstream passes the reply straight to `bash -c` and scores a fenced or chatty answer as the shell failure it becomes, so stripping fences would repair the thing being measured. The shared base pins the two vendored copies to each other at run time. Every response echoes a digest over core.py + scenarios.py + shellesc.py, and the task raises rather than scoring when it does not match what the prompts were built from — drift would otherwise be silent, with prompts from one fixture set, grading against another, and every number still plausible. An evaluator that omits the field is treated as a mismatch, not as no objection. `report.json` publishes upstream's own `pass_rate_pct` as the headline, with `score` riding as an alias on the same interval rather than a second one computed the same way; all seven failure classes are columns even at zero, plus `n_unknown_class` so a taxonomy upstream widens is counted rather than dropped, and per-tier / per-scenario rates carry no interval because each sits on a different population from the headline's. The acceptance anchor is a standing test now: the release recompute needs only the arm file, and the replay needs a running evaluator and agrees with upstream's recorded GNU verdicts 224/224 on both `passed` and failure class. Both ship `experimental`: the shipped path grades inside a GNU userland image that has not been built or run, so no score impact is quantified against it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…iding CI failed on the two acceptance tests with FileNotFoundError while 6706 tests passed. The arm file was never in the commit: `.gitignore` carries a bare `data/`, which matches at any depth and so swallowed `tests/acceptance/quotebench/data/` along with the run-artifact directories it was written for. `git add tests/acceptance/quotebench` reported only the README and the test, and I did not check the staged list against what the tests read. Re-included with an explicit negation. Both lines are needed: git will not descend into an excluded directory, so the directory has to be re-included before anything inside it can be. Scoped to `tests/acceptance/*/data/` rather than to this one path, because any acceptance fixture would hit the same rule. The failure mode is the reason to commit the fixture rather than fetch it: a test that reads a checked-in file passes locally and fails only in CI, which is the slowest possible place to learn the file is missing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Three behaviour changes, five corrections. No verdict moves: the released rollout grid still replays 224/224 on `passed` and on failure class. Grading no longer blocks the event loop. `execute_quotebench` is fully blocking and `evaluate` is `async def`, so FastAPI ran it ON the loop rather than in the threadpool it gives a plain `def`. Measured with four concurrent gradings of a `sleep 3` reply while polling `/health`: wall clock for the four 12.02 s -> 3.01 s /health polls served 2 -> 59 /health worst latency 11969 ms -> 1.9 ms The stall was not confined to quotebench -- one shared loop, so a slow Bash reply also held up LiveCodeBench and HumanEval on that worker. The other three sources were already off the loop, two of them via `asyncio.to_thread`, so this is the package's own idiom. The digest handshake now settles in `setup()`, over a new read-only `GET /quotebench/digest`. It was checked only per sample in `feedback`, which `TaskRunner` catches and turns into a FAILED context -- so a skewed evaluator did not abort the run, it failed all 56 samples and wrote a report.json reading 0.0. On a hard shell-quoting benchmark that is a number someone would act on. `setup` is the only stage that can stop a run; checked there, a skew costs one GET and no inference. The per-sample check stays as a second line, for an evaluator replaced mid-run behind one address. The URL is joined relatively so a service mounted under a prefix keeps it. `n != 1` is refused at construction instead of being silently narrowed. `report()` reads one rollout per sample on every axis it publishes -- the headline, the seven failure-class counts, and the per-tier and per-scenario rates -- so a larger budget was generated, graded, paid for and then dropped behind a score that still looked like a complete run. Averaging the extras in is a design question, not a fix: a failure class has no agreed reading across repeats, and upstream's anchor grid is single-draw, so there is no reference answer to port. Both leaves' reference_impl.notes say so. Corrections: - The acceptance README claimed `crossover.py` carries a transport-alias map. It does not; `rollouts.py` reads the released spelling only as dict keys over stored verdicts, and both modules that build a command reject it. - VENDORED.md described the 56-oracle check and the 224/224 replay as one verification level. They are gated differently, and the stronger of the two is the one CI does not run. - The README and Dockerfile told operators to run the service itself with `--network none`, which cannot work -- it has to be reachable to be called. Both now state the two real isolation models: upstream's `exec_docker` enforces `--network none` per attempt in its own argv, and the in-container route needs an internal network, not no network. - Attribution line normalized to the form the other twelve modules use. - Restored upstream's `__version__` in the partially-vendored `__init__`, so the two copies of the package do not answer differently. Not added to the digest, which covers only what decides a verdict. Tests: the setup guard aborting rather than degrading to 0.0, the 404 and unreachable paths, the relative URL join across five base shapes, and the `n` refusal. Plus the contrast the guard exists for -- an all-fails run publishing a believable zero. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…un and measured Both tasks shipped `experimental` on one sentence: the GNU userland image the shipped path grades in had not been built or run, so no score impact was quantified against it. It has now been run, and the sentence no longer holds. No Docker daemon is available here (container-in-container: AppArmor refuses the legacy mount(2) podman/crun/bwrap need), so the Dockerfile's layers were replayed into a udocker/PRoot container. The base is pulled BY DIGEST, which is load-bearing: `debian:stable-slim` has moved since the Dockerfile was written, so a tag pull grades in a different userland. The pin is not cosmetic either -- an ordinary host here has mawk and no gawk at all, against the image's gawk 5.2.1. The anchor reaches 224/224 on `passed` and on failure class in BOTH userlands, so the host and containerized executors agree with upstream and with each other: the userland pin moves no verdict on the anchor data. `docker build` itself stays unexercised, and the notes say so. Live crossover through that container, gpt-5.5 -- the one published row with released rollouts, and both system prompts are byte-identical to the released ones: 96.4 / 30.4 / 44.6 / 98.2 against a published 100.0 / 28.6 / 50.0 / 89.3. Three cells sit inside sampling noise; NN is +8.9 at z~2.0. All 19 per-task disagreements were attributed by re-grading upstream's OWN stored reply through the same container, which reproduces upstream's verdict and failure class 19/19 while our reply text differed 19/19 -- the divergence is the model, not the port. The status test is rewritten rather than deleted: it now pins that a `stable` whose notes stop naming the containerized anchor fails, so promoting the status and dropping the evidence cannot be separate edits. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Eight findings from a second review pass. No scoring behaviour changes: the 224-execution anchor still reproduces upstream's recorded GNU verdicts on both axes -- through the HTTP path, and now also without a server. **The anchor becomes a CI gate.** It was enforced by hand only: `tests/acceptance/quotebench/` skips when nothing is listening, so what CI actually ran was 56 oracles under the `raw` contract plus string-level `transport()` assertions. None of that grades a real model reply, exercises the nested transport end to end, or touches the failure taxonomy. The new `tests/unit/vendor/code_evaluator/test_exec_quotebench_anchor.py` replays the whole 224-execution crossover grid through `execute_quotebench` -- stdlib plus the vendored package, no `fastapi`, ~0.7s -- reading the acceptance fixture rather than copying it, so there is still one arm file and one hash pin. Measured rather than asserted: a wiring bug that ignored `contract` and graded every nested sample as raw passes the entire suite green today, and lands at 158/224 against the grid. **Per-tier and per-scenario rates now share the headline's denominator policy.** They divided by finals only while the headline divided by requested, so with `fails > 0` every row read HIGHER than the score above it -- and `feedback` raises on an evaluator or protocol fault, so a partial outage is exactly what produces failed samples, and exactly when someone reads those rows. A failure is now charged to its own tier and scenario; one that lost its `raw_sample` is charged only to the headline, rather than pooled into an axis it never had. Also: - `scenarios_digest()` is cached. It re-read and re-hashed three files on every verdict, from the event loop -- `evaluate` is `async def`, which is the same hazard the `asyncio.to_thread` call two functions away exists to avoid. - An unknown `task_id` names the field and the frozen core's size instead of reaching `msg` as a bare `'no-such/task'`. The handler unwraps `KeyError.args[0]`, since `str()` on a `KeyError` is the repr of its argument and was re-quoting the sentence. - The acceptance reachability probe asks for `/quotebench/digest` rather than `/health`. The latter is source-agnostic by design, so any unrelated service bound to the port defeated the skip and the replay died partway through on a 404 that read like a grading failure. - The unit tests no longer inherit `SIEVAL_CODE_EVAL_API` from the environment. Exporting it -- which the acceptance instructions tell you to do -- turned the handshake URL assertion red for a reason it did not name. - `VENDORED.md` records that the flat `ResourceMetrics` widens every other source's response: the six task modules that bucket unrecognised fields into a `resources` catch-all (HumanEval x2, LiveCodeBench x2, MBPP, MultiPL-E) gain four always-null keys in every rollout record. No score moves, and each alternative (per-source model, `response_model_exclude_none`) breaks something else. - `tests/README.md` records the `tests/unit/vendor/` carve-out and scopes it to patches `VENDORED.md` marks *in-tree by decision*, so these tests are not removed the way the earlier upstream-bound ones were. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…LFA landed This PR said it owed this: "Neither is merged yet — whichever lands second should reconcile the two rather than grow a third shape." NL2SH-ALFA (#139) landed first, so QuoteBench is second and the debt is due. Reconciling does NOT mean merging them. The split is forced by the benchmarks, not chosen: `/shell-evaluations` returns facts because deciding Bash equivalence needs an embedding model and the service holds no credentials, while `quotebench` returns a verdict because an exact final-state comparison is decidable in-process. One is stateful and pinned to five prepared images at `--workers 1`; the other is stateless with a per-attempt fixture and is safe in parallel. Folding either into the other would cost exactly the property that makes it work. So VENDORED.md gains a table of the two and a rule for picking between them, phrased as a row to match rather than a column to add. One concrete interaction, not just prose: `quotebench` is a FOURTH route that needs `/tmp`, which the NL2SH images delete on their first `git clean -fd`. That entry listed three (`exec_py_code`, `exec_js`, `exec_ts`) because it was written before this source existed. `run_attempt` calls `mkdtemp` per attempt, so on those images grading raises before any command runs — loud, but it reads as a broken grader rather than as a route on the wrong image. Also corrects the `tests/unit/vendor/` carve-out added earlier in this branch. It claimed an upstream-bound patch keeps its tests in `scitix/code-evaluator`; NL2SH-ALFA in fact keeps 26 of them in `vendor/code-evaluator/tests/`, a third location that did not exist when the carve-out was written. The note now states both locations and the trade between them — tests that travel with the code are outside CI's collection, which is the price of their travelling. Verified on the rebased tree rather than assumed: all five routes are live on one merged server (`/shell-evaluations` answers 422, not 404), the digest is unchanged, and the 224-execution anchor still replays 224/224 through the merged HTTP path. NL2SH's own 53 tests (27 task + 26 vendored) pass. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
ethan-scitix
force-pushed
the
feat/quotebench-port
branch
from
September 11, 2026 04:14
e3738ae to
bf91a83
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
Summary
LeonardNJU/quoteBench@693325a6, Apache-2.0): 56 one-shot Bash tasks over 14 operation families, graded by the exact final program state — file bytes, argv, JSON, directory contents or Git history — never by comparing a command string.rawexecutes the reply verbatim;nested_shellinterpolates it intobash -c "R"unescaped. Upstream's finding is that two matched command paths score alike while reversing the order of 5 of 26 model pairs (19.2%) — so one number for "QuoteBench accuracy" would hide which command path produced it, and the registered name is the only place that distinction can live.quotebenchsource invendor/code-evaluator, whose unit of work is a task id plus one command rather than a program plus test cases: the task builds its own filesystem fixture and there is no reference command to compare against, so none of the existingexec_*modules could carry it. sieval itself still executes nothing.passedand 224/224 on failure class, and the release recomputes upstream's published gpt-5.5 crossover row to the decimal. A live gpt-5.5 run then covers what a replay cannot — the prompts and the model call — landing at 96.4 / 30.4 / 44.6 / 98.2 against a published 100.0 / 28.6 / 50.0 / 89.3, with every one of the 19 per-task disagreements attributed to reply text rather than to grading.stable. They wereexperimentalon a single sentence — the GNU userland image the shipped path grades in had not been built or run — and that has now been done. The anchor reaches 224/224 inside the pinned image as well, so the host and containerized executors agree with upstream and with each other: the userland pin moves no verdict on the anchor data.docker builditself stays unexercised (no daemon on this box); the image was materialized under udocker/PRoot, pulled by digest, becausedebian:stable-slimhas moved since the Dockerfile was written and a tag pull would grade in a different userland.Test Plan
Automated
ruff check && ruff format --check) — 563 filesty check)tests/unit, plus 3 intests/acceptance/quotebench, the latter run against the evaluator inside the pinned imagemain@e55faa35. Two benchmarks have now landed in the same threevendor/code-evaluator/files — MultiPL-E (feat(tasks): add MultiPL-E — HumanEval + MBPP in 24 languages #138) and NL2SH-ALFA (feat(tasks): add NL2SH-ALFA — execution-graded natural language to Bash #139) — and every conflict was additive both times: sources each registering in one dispatch chain and documenting it in the same two files, resolved by keeping both sides. The merged server was probed live rather than assumed to have survived the auto-merge: five routes coexist (/health,/languages,/shell-evaluations,/quotebench/digest,/evaluations),/shell-evaluationsanswers 422 rather than 404, thescenarios_digestis unchanged, and the 224-execution anchor still replays 224/224 through the merged HTTP path. NL2SH-ALFA's own 53 tests (27 task + 26 vendored) pass on the rebased treecheck_preflight.py— 25 PASS, 0 FAIL;check_links --level deep88/91 reachable (the 3 exceptions are pre-existing StackOverflow/Wikipedia 403s)Manual
Alignment here is a per-prediction check rather than an aggregate comparison, because upstream releases rollouts and not just scores. The replay anchor below needs no model and no container at all; a live gpt-5.5 run through the pinned image was then added to cover the half a replay cannot reach — the prompts, the model call, and the deliberate absence of extraction:
Replay anchor, through the shipped HTTP path (
TestClient, so pydantic validation and the declared response model are both in play). Upstream'sraw-vs-nestedarm carries four executions per record,{bsd,gnu} × {raw,nested}, GNU flaggedreplay: true— the full crossover from stored replies:passedagreementThe published table is reproducible from the release — recomputed from the arm file and matching upstream's README row for gpt-5.5 to the decimal: RR 100.0 / RN 28.6 / NR 50.0 / NN 89.3; damage −71.4, compensation +60.7, matched gap −10.7. Both of these are standing tests under
tests/acceptance/quotebench/.All 56 oracles pass through the HTTP path.
Validity screened before any code was written, against the BashBench post-mortem (a "functional correctness" metric can be causally independent of the model's answer). On the pinned tree: oracle 56/56, a valid no-op
:0/56 (allsilent-wrong), and upstream's naive probe 14/56 — exactly the 14 tier-0 benign controls, set equality verified. The metric reads the prediction.Bound evidence (
sieval/tasks/CLAUDE.mdrequires it for an execution path): per-attempt wall clock is 2 ms median, 13 ms p95, 17 ms max, the whole 56-task core sweeping in 0.2 s — roughly 880× headroom under upstream's 15 sTIMEOUT_S. No bound binds on the frozen core.The image, built and run — this is what promotes the status.
QUOTEBENCH_EXECUTORdefaults tolocal, which reproduces the GNU verdicts on an ordinary Linux box; the shipped path instead grades insidedocker/Dockerfile.quotebench, and that userland is now exercised. No Docker daemon is available (container-in-container: AppArmor refuses the legacymount(2)podman/crun/bwrap need), so the Dockerfile's layers were replayed into a udocker/PRoot container from the pinned base by digest —debian:stable-slimhas moved to index0463…since the Dockerfile pinned328d…, so a tag pull would silently grade elsewhere. The pin earns its keep: the host measured here ships mawk with nogawkat all, against the image's gawk 5.2.1. Inside it the anchor is again 224/224 onpassedand 224/224 on failure class, so host and container agree with upstream and with each other — the userland pin moves no verdict on the anchor data. What is still not exercised isdocker builditself and a real daemon's namespace isolation; the notes say so rather than claiming the packaging.Live alignment, gpt-5.5 through that container — the one published row with released rollouts, so the comparison is per-task and not only aggregate. Both system prompts are byte-identical to the released ones (792 / 877 chars), the user message equals the dataset instruction 112/112, and the run's prompt-token range (209–353) equals the arm's exactly.
n=1, no temperature set (upstream records none).fails=0,n_unextracted=0,n_truncated=0, zero anomalies. The two tasks give the matched diagonal; the off-diagonal is upstream's own construction, the same reply through the other transport, replayed across both tasks' persisted finals:damage −66.1 (published −71.4), compensation +67.9 (+60.7), matched gap +1.8 (−10.7).
Every disagreement attributed, per prediction. Across the four cells, 19 tasks disagree with upstream's recorded verdict. For each, upstream's own stored reply was re-graded through the same container: it reproduces upstream's verdict and failure class 19/19, while our reply text differed 19/19. The clearest case is NN, where upstream's four
shell-syntaxfailures all reproduce on upstream's replies and none occur on ours. The divergence is the model, not the port.Two caveats kept rather than buried: the matched gap flips sign on the strength of NN alone, and
nis pinned to 1 by design, so repeats cannot sharpen a 56-sample single-draw grid — only a second published model can. The gateway also reportsgpt-5.5-2026-04-23where upstream records onlygpt-5.5; same name, unverifiable snapshot.Notes for review
Which prompt backs the published column was measured, not assumed. The nested task ships
SYSTEM_PROMPT_NESTED_SHELL_V2because all 56 stored system messages in the released arm match that constant by identity. The v1 constant the same module still ships adds a one-line-reply rule and an escaping tutorial, and upstream marks it contract-confounded.No extraction, deliberately. Upstream passes the reply straight to
bash -cand scores a fenced or chatty answer as the shell failure it becomes. Stripping fences would repair the thing being measured.Two vendored copies, pinned to each other at run time. The evaluator needs
setup/check/harness; sieval needs only the prompts. Every response echoes ascenarios_digestovercore.py+scenarios.py+shellesc.py, and the task raises rather than scoring on a mismatch — including when the field is absent, which is what an evaluator predating this source looks like. Drift would otherwise be silent: prompts from one fixture set, grading against another, every number still plausible.An upstream gap the port has to own. The released records spell the transport
nested, butpublic_cli.command_for_transportaccepts onlyraw/native/nested-shelland raisesValueError: nested— sopython -m quotebench scorecannot read upstream's own release. Both modules that turn a contract name into a command reject the released spelling:public_cli.command_for_transport, andcrossover.canonical_contract, which raisescontract crossover supports raw/nested-shell records, got 'nested'. (rollouts.pydoes read the released spelling, but only as literal dict keys over already-stored verdicts — it never turns a name into a command, so it needs no mapping.) The name → transport mapping therefore lives inapp/exec_quotebench.py.This is a second, differently-shaped shell route, and the reconciliation is paid. NL2SH-ALFA's
/shell-evaluations(#139) landed first, so this PR is the one that lands second and owes what this paragraph used to promise. It is paid inVENDORED.md, as a table of the two routes plus a rule for picking between them — not by merging them. The split is forced rather than chosen:/shell-evaluationsreturns facts, because deciding Bash equivalence needs an embedding model the service holds no credentials for, whilequotebenchreturns a verdict, because an exact final-state comparison is decidable in-process. One is stateful and pinned to five prepared images at--workers 1; the other is stateless with a per-attempt fixture and safe in parallel (measured: 672 executions at 32-way, no verdict moved). Folding either into the other costs exactly the property that makes it work.One concrete interaction fell out of that, not just prose:
quotebenchis a fourth route needing/tmp, which the NL2SH images delete on their firstgit clean -fd. That entry listed three, because it was written before this source existed —run_attemptcallsmkdtempper attempt, so on those images grading raises before any command runs, which reads as a broken grader rather than as a route on the wrong image.A structural fact worth knowing when reading the fixtures: only 48 of the 56 instructions carry the U+27EA/U+27EB literal-text markers. The 8 that do not are exactly the
find-globandbulk-renamefamilies, whose hazards live in the fixture as hostile filenames already on disk rather than in a string handed to the model. The identity test pins that split rather than the bare count.Review fixes (commits
bc37c5fd,bf91a83f)Eight review findings, all addressed. No scoring behaviour changes; the anchor
still reproduces 224/224 on both axes, through the HTTP path and now also
without a server.
The 224-execution anchor is now a CI gate. It was enforced by hand only:
tests/acceptance/quotebench/skips when nothing is listening, and the in-CIchecks covered 56 oracles under the
rawcontract plus string-leveltransport()assertions.tests/unit/vendor/code_evaluator/test_exec_quotebench_anchor.pyreplays the whole grid through
execute_quotebench— no HTTP, nofastapi,~0.7 s. Measured gap it closes: a wiring bug that ignored
contractand gradedevery nested sample as raw passed the entire suite green, and lands at 158/224
against the grid.
Per-tier / per-scenario rates now share the headline's denominator policy.
They divided by finals only while the headline used requested, so with
fails > 0every row read higher than the score above it — and sincefeedbackraises on an evaluator fault, an outage is exactly what producesfails. A failed sample is now charged to its own tier and scenario; one that
lost its
raw_sampleis charged only to the headline rather than pooled intoan axis it never had.
scenarios_digest()is cached — it re-read three files on every verdict,from the event loop.
An unknown
task_idsays what is wrong. It reachedmsgas a bare'no-such/task'; now it names the field and the frozen core's size. Thehandler also unwraps
KeyError.args[0], sincestr()on aKeyErroris therepr of its argument and was re-quoting the sentence.
The acceptance reachability probe asks for
/quotebench/digest, not/health./healthis source-agnostic, so any unrelated service bound tothe port defeated the skip and the replay died mid-run on a 404 that read like
a grading failure.
The unit tests no longer inherit
SIEVAL_CODE_EVAL_APIfrom theenvironment — exporting it (as the acceptance instructions tell you to) turned
the URL assertion red for a reason it did not name.
VENDORED.mdrecords that the flatResourceMetricswidens every othersource's response: the six task modules that bucket unrecognised fields into
a
resourcescatch-all (HumanEval ×2, LiveCodeBench ×2, MBPP, MultiPL-E) gainfour always-null keys in every rollout record. No score moves; it is a
record-shape change, and the alternatives (per-source model,
exclude_none)each break something else.
tests/README.mdrecords thetests/unit/vendor/carve-out and scopes itto in-tree by decision patches, so these tests are not removed the way the
earlier upstream-bound ones were.
Reconciled the two shell routes (
bf91a83f), now that NL2SH-ALFA haslanded and this PR is the one arriving second — see the Notes section above.
Also corrects the
tests/unit/vendor/carve-out added earlier in this branch:it claimed an upstream-bound patch keeps its tests in
scitix/code-evaluator,but NL2SH-ALFA keeps 26 of them in
vendor/code-evaluator/tests/, a thirdlocation that did not exist when the carve-out was written.
Commit trailers normalised. The range carried the same co-author under two
spellings; since the rebase re-SHA'd every commit anyway, the odd one was
folded in at zero marginal cost. Content is byte-identical across the rewrite
(verified by diffing the trees), and
backup/pr140-pre-rebase-e3738aebonorigin pins the pre-rebase state.
Checklist
Required (all PRs)
type(scope): description)AI-Generated Code - <model> (<provider>)in module docstring — on every new first-party module; the vendored upstream files are exempt and stay byte-identicalcore/If: New or Modified Benchmark
quotebenchis code-generated, so it follows thelocal:bring-your-own convention (scripts/gen_quotebench_snapshot.pywrites into the data dir; nothing is committed).sieval dataset downloadcorrectly reports the BYO requirement rather than silently succeeding, and a test pins the staged rows againstall_tasks()becauselocal:sources cannot declare a checksum.__init__.py— discovery is by AST scan, so registration is automatic;__init__.pyiandsieval/meta/index.jsonregenerated by their sync scriptsIf: community/ Changes
sieval/community/quotebench/is byte-identical to upstream at693325a6, pinned by sha256 taken from the upstream tree rather than from the copies, and proven to bite by reverse mutation (substituting an ASCII lookalike for the U+27EA marker turns 2 of 7 tests red). Onlycore/scenarios/shellescare vendored here —harnessis deliberately left out, since vendoring it would put an executor one import away from a task.If: New Dependency
httpxclient to the existing code-evaluator service.🤖 Generated with Claude Code