feat(tasks): add NL2SH-ALFA — execution-graded natural language to Bash

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• NL2SH-ALFA: dataset nl2sh_alfa + two tasks — nl2sh_alfa_0shot_gen (upstream's Base reading) and nl2sh_alfa_0shot_gen_parse (its shipped prompt + parse_bash) — graded by executing the model's Bash command and upstream's gold against a prepared filesystem. Paper: arXiv:2502.06858 (NAACL 2025); harness: westenfelder/InterCode-ALFA (icalfa 0.3.6, the version the paper states Table 5 was produced with); dataset: westenfelder/NL2SH-ALFA. Both MIT.
• The scored set is 300 rows, not the 600 the card advertises (600 counts two verified commands per instruction). This is a live trap, not bookkeeping: index_to_img raises past 299 and submit_command swallows that into a score of 0 after a bare print, so a caller who believes the larger figure loses half a run silently.
• Grading reads the harness's own vendored gold table, not the Hub's bash column — that is what upstream scores, and the two disagree on 2 of 300 rows (index 38 echo -n 'hello' | base64 vs echo 'hello' | base64, i.e. aGVsbG8= vs aGVsbG8K; index 100 awk 'length < 20' vs < 40), with 3 more differing in wording. The dataset exposes both sides, joined positionally under a 300/300 assertion, so the mismatch is visible in the data instead of resolved inside the grader.
• _parse is a new variant word in sieval/tasks/CLAUDE.md, licensed by upstream publishing two protocols over one benchmark and one grader — not by a preference between them. The bare name goes to Base, since parsing is one of four methods the paper layers on the benchmark. Their gap reaches 32% on small models, so one number would hide whether a model was scored on translation or on formatting compliance.
• New infrastructure: vendor/code-evaluator gains its first stateful route (POST /shell-evaluations) and five images, because the baseline filesystem is the image. Each Dockerfile.nl2sh-{1..5} is upstream's verbatim (verified line-by-line, not asserted) with a service block appended under /opt — which upstream's own docker.gitignore excludes, so the baseline commit cannot see it and git clean -fd cannot delete it mid-run.
• New dependency nl2sh-alfa = ["scipy>=1.16.3"]: 1 - scipy.spatial.distance.cosine(...) is the FEH's exact expression, so it is the metric and not a convenience — a hand-rolled dot product could disagree with upstream on a pair sitting within float noise of the 0.75 threshold. No lighter alternative that keeps that guarantee; scipy is already resolved in three existing groups.

Two upstream behaviours reproduced deliberately

• Upstream's own quoting rewrites 37 of its 300 golds. clean_cmd wraps a command in double quotes and hands the string to docker-py, which normalizes it through shlex.split (docker/utils/utils.py:486, verified at 7.1.0). Index 230's awk '{print $2 " " $1}' is cut at the quoted space and reaches bash as a syntax error, making that gold's "correct answer" an awk error message. Writing the obviously-correct [entrypoint, "-c", command] would disagree with every published number on 12% of samples, so command_argv reproduces the transform and a test pins it over all 300 golds.
• md5deep is absent from every image, and upstream's get_hash_cmd still picks it for any path without a . — so both sides get the same failure text, compare equal, and a directory-only change always scores full credit on part 2. Preserved; a test asserts the equality, which is the property the metric reads.

Related Issues

Refs the NL2Bash scoping that preceded this: NL2Bash was declined because its headline is human-judged and its grader is GPL, and NL2SH-ALFA was named as the set to land once stateful bash execution existed. This PR is that execution plus the port.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check)
• Unit tests pass — 6770 passed, including 51 new sieval-side tests; plus 17 in the vendored service (vendor/code-evaluator/tests/test_exec_sh.py, which drives the whole reset/status/hash/timeout/refusal protocol against a throwaway git tree via NL2SH_FS_ROOT — no container needed)
• scripts/check_preflight.py — 25 checks, all PASS
• Rebased onto main (2cdd39f8); the pdm.lock conflict was resolved by re-locking, not by picking a side, and the resulting diff against main is 4 lines (groups list, content_hash, numpy/scipy group membership) with zero version drift

Manual

• sieval dataset download nl2sh_alfa succeeds (2.5 MB, 5 files, revision-pinned)
• sieval task show nl2sh_alfa_0shot_gen / _parse and sieval dataset show nl2sh_alfa all resolve by name
• All 11 vendored upstream files (5 gold tables, 5 setup scripts, docker.gitignore) verified byte-identical to upstream after commit by sha256 — the pre-commit hooks include end-of-file and trailing-whitespace fixers, so "hooks passed" is not the same evidence
• Each of the five Dockerfiles verified line-by-line against upstream's at the pinned commit; the only difference is the documented docker/nl2sh/ COPY prefix, and ENV FILES appears on fs1 only — which matters, because exactly one gold (in fs1) reads $FILES
• Score comparison table — NOT included, and cannot be. Building the five images and running the validation ladder needs a host with Docker; the dev box has none (no docker/podman/nerdctl, and bwrap cannot unshare). This is why both tasks ship status="experimental". The ladder, for whoever has such a host: (1) feed each gold back as its own prediction — must be 300/300; (2) run upstream icalfa at eval_mode="embed", eval_param=0.75 over one stored prediction set and compare per-sample verdicts, the only evidence the topology divergence below is harmless; (3) align against gpt-4o-2024-08-06 → 0.74 and gpt-4o-mini-2024-07-18 → 0.71 (σ ≈ 2.5 pp at n=300, so a 3 pp band). Rungs 2 and 3 are what promote off experimental; rung 1 alone does not, since it never exercises the FEH.

Checklist

Required (all PRs)

• PR title follows conventional format
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/ — core/ is untouched
• Deleted code verified — nothing deleted; the three existing /evaluations backends are untouched, so no existing task's grading moves

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table — see Test Plan → Manual for why this is blocked on a Docker host rather than on this diff
• Dataset loading tested (sieval dataset download nl2sh_alfa succeeds)
• Task registered — tasks/__init__.py discovers by module and both .pyi stubs plus sieval/meta/index.json are regenerated (57 datasets / 72 tasks); verified by re-running both sync scripts after the rebase and getting zero diff, since an auto-merged index can be stale without conflicting

If: community/ Changes

• Upstream diff documented — sieval/community/intercode_alfa/'s docstring states what is verbatim (index_to_img, parse_status, get_hash_cmd, clean_cmd, parse_bash, the three reward parts with their erf/round shapes, the A/??/C filter, the 1000-char truncation, the strict > threshold) and what is left out and why (everything touching Docker or a model: execution belongs to the code-eval service, the embedding call to the task, which owns credentials and records). Three upstream behaviours that read like defects are called out individually with the reason each is preserved.
• License attribution preserved — both upstreams MIT, cited with pinned commits (icalfa 2d3a6947, NL2SH b405201a)
• vendor/code-evaluator/VENDORED.md updated with the new patch, its two deliberate divergences, and the standing "land upstream in scitix/code-evaluator and re-vendor" item

If: New Dependency

• Added to correct PDM dependency group (nl2sh-alfa, and both tasks name it as deps_group; declared by hand because check_dep_coverage scans only tasks/ and datasets/ and this import is reached through community/)
• Justified in Summary

Three divergences from upstream, each in reference_impl.notes

1. One container with a reset before each command, instead of two peer containers. Upstream runs the gold in a second container so the model can never perturb it. Here the gold runs first, on a freshly restored tree, which preserves that property within a sample and yields exactly the inputs p1/p2/p3 need. What is not preserved is cross-sample residue in gitignored paths — and upstream's own containers persist across all 300 samples, accumulating the same.
2. The 10 s command wall (utils.TIMEOUT_DURATION) is applied to the gold as well. Upstream bounds only the model's command; an unbounded gold hangs a service rather than one sample. This is an execution-safety bound, so it owes evidence it never binds — n_gold_timeouts is published for exactly that.
3. The embedding is served over an OpenAI-compatible endpoint rather than Ollama's /api/embeddings, so the weights are the Hub's rather than Ollama's f16 GGUF conversion of them. Both calls stay separate, as upstream's are. Quantifiable by re-scoring one stored run through both endpoints and counting verdict flips.

Also worth a reviewer's attention: parse_status raises IndexError on an odd token count (a path containing a space), where upstream swallows it into a score of 0. This port lets it propagate per .claude/rules/tasks.md, so the sample lands in fails with a recorded reason — and under DENOMINATOR_REQUESTED a fail is already charged as wrong, so the headline agrees with upstream's 0 while the cause stays visible instead of being indistinguishable from a model that answered incorrectly.

🤖 Generated with Claude Code