fix: env and importer reward integrity

deepgym/CHANGELOG.md

@@ -22,6 +22,7 @@
 - **Traces keep every action, every grade, and the code and image that made them (schema 1.1).** A model turn that issued two actions (two commands, or a tool call and a command) dropped the whole trace from `--rows rl` exports, and ATIF split it into one agent turn per action. Step each action and pass the turn's messages with its first one only: `Trace.turns` groups them (a 1.0 trace, which used empty messages for "not recorded", keeps one turn per step), RL rows keep the trace and gain `steps` (every action with its observation text and screenshot, done flag, exit code, error, step reward and the `message_index` of the reply that produced it), and ATIF writes one agent step per turn with a tool call and a result per action, the step reward and step verifier output now in each result's `extra`. Calling `score()` twice kept only the last grade; earlier ones are now in `prior_results`, and `deepgym traces export` refuses two copies of a trace that disagree instead of keeping one. `Trace.to_trajectory()` keeps an action the session applied even when its screenshot then failed (it dropped the action and its reward), and totals step rewards the way the session does (`sum()` rounds differently from Python 3.12 on). Provenance adds `deepgym_commit`, the git commit of the deepgym checkout (null for an installed package), with `deepgym_dirty`, set if the package folder's files differ from that commit or changed after deepgym was imported, and the image the sandbox booted with the digest its provider resolved: Apple container and Docker from the container, Daytona from its snapshot (the default or a named one). Where there is none, as for an image Daytona builds, Modal, E2B or the local backend, `digest_unavailable` says why. Schema 1.0 traces still load and export.
 - **Core reward path: each failure lands on whoever caused it.** A verifier that prints a score and then crashes no longer keeps that score, in any verifier shape; before, Python's exit 1 for an uncaught exception read as the protocol's "failed" and the printed partial score reached the reward function. An exception hook now decides, while the script keeps running as `__main__` with its own `__file__`: before the policy's output is in play, the crash is the verifier's own failure, with no verdict (`runtime_error`, returned rather than raised); once it is (a solution process started, or the verifier opened the solution's file, or a session verifier opened a file in the agent's work directory, so running the solution with `exec` and parsing malformed policy output count), the crash came from handling it, so it is a judged 0 with `error_type='solution_error'`, a new model fault. So is a result the verifier can't normalize (NaN, not a number) after that point. Whatever the solution does (raise, exit, hang, die, return garbage, print unparseable output) is therefore its own 0 even when the verifier doesn't handle it; before, an uncaught solution error became a verifier error with no verdict, as `import_error` or `timeout` when the solution raised `ImportError` or `TimeoutError` itself, so a solution could dodge its score by breaking the verifier. Every failure from the solution process now reaches the verifier as a `deepgym_solution.SolutionError`, still of the builtin type it raised (`except ValueError` and `except TimeoutError` keep working). A solution that runs past its time limit is a judged 0 with `error_type='solution_timeout'`, a new model fault, on every path; `timeout` stays DeepGym's own timeout, with no verdict. Load test data before the solution: a verifier that fails after starting it scores 0. `LaunchError` is now a `BaseException`, so a verifier's `except Exception` can't score the machine's failure; uncaught, it ends the run with `sandbox_error` (no verdict). Before the first solution start, a preflight checks free disk and inodes and that the isolation launcher (sudo, setpriv, bwrap, sandbox-exec) starts code from a fresh directory, and a judge-side failure while preparing the run (a temp file, the loopback relay) is a `LaunchError` too. Once solution code has run, a failed relaunch is the solution's, a `SolutionError` scored 0, since it may have used up the disk or processes. The judge prints a per-run token on stdout once the policy's output is in play. A single-turn sandbox that dies after that token (its Docker or Apple container stops, or a Modal exec ends with the sandbox finished, which return what was printed) is the policy's doing: a judged 0 with `error_type='policy_sandbox_failure'`, where it was `sandbox_error`; so is a run that outruns its time limit after the token, locally or in a sandbox (`solution_timeout`), such as a verifier regex the solution's output made backtrack. One that dies before the token (creation, setup, uploads, the verifier loading its own fixtures), one whose provider returns nothing printed (Daytona, an E2B or Modal call that raises), or one Daytona evicts or times out in its queue (as its error or its refreshed sandbox record says) keeps no verdict. Wrappers report it as the new `deepgym.exceptions.SandboxDiedError` (also in `deepgym.sandboxes`), which `is_infrastructure_failure` never counts as the provider's failure. Every verdict line the harness prints starts a fresh stdout line, so a partial line the verifier left unterminated can't hide it, and a stderr the solution broke (a full disk under a file-backed stderr) can't change the verdict or the exit code. Session verifiers use the same guard, and the code they load gets the step's time limit as its deadline, as the judge does, with a few seconds' grace for the verifier to report. A session step that fails after its action may have run (the provider failing while it applies the action, takes the screenshot or runs the step verifier, not only a `VerifierError`) now ends the episode: the action stays on the trajectory with no step reward, further steps are refused, and `score()` returns no verdict for a network or provider failure (`is_infrastructure_failure`) or a failed step verifier; anything else follows from the agent's actions (its sandbox died, a write failed, its files, a folder or a planted link, blocked the step-output file) and is a judged 0 (`error_type='policy_sandbox_failure'`). A sandbox that dies under the final verifier is a judged 0 too, once the agent has acted. `run_episode` returns the result a failed step fixed instead of raising. Before, the session stayed runnable, so a retry applied a second action past `max_steps`, scored it, and the first action was never recorded. The new observation is now taken before the step verifier runs, so a failed screenshot no longer leaves a recorded reward and a stale observation behind for a retry to replay, and trace replay (`Trace.to_trajectory()`) keeps a recorded step that failed without an observation; trace capture also keeps a step interrupted after its action ran. A `write_file` blocked by the agent's own files (a folder or link in the way, found before anything is written) raises `ActionError` (rejected, the episode goes on); any other write failure is a `SandboxError`. The judge's setup refuses an existing `dgsolver` account that is root or shares its uid, gid or groups with another account (the check session setup already ran, now shared and also refusing uid or gid 0), and the judge never drops the solution to uid or gid 0; before, an image with a root-alias `dgsolver` ran the solution as root. Nothing a solution starts outlives its verifier any more: with its first solution the verifier starts a watcher process, in its own session, that kills every solution process group (a `load_solution` server, a `run_solution` script and whatever they started) once the verifier ends, however it ends (a host timeout or cancel, a crash), even when the solution hogs the interpreter in a long C call; a `run_solution` script waits to start until the watcher knows it, and a verifier's own exit waits for the watcher, so a sandbox user's sweep never reaches the next grade (on macOS, a child that detaches into a session of its own still escapes, which is why subprocesses need `allow_subprocess` there); before, such a server, or a busy child a solution started, kept running after the run. A cancelled or timed-out local run, sync or async, kills its verifier's whole process group and reaps the verifier, so neither a verifier's child nor one holding its output pipes outlives or stalls it, and a traced session's `close()` retries a sandbox delete that failed instead of returning silently.
 - **SWE-bench Pro / MiMo patch runner: two cases where a correct patch got no grade.** In a task whose `repo_dir` is a subdirectory of the repository, the hidden test files were saved and restored at the wrong path, so a patch that also edited a hidden test kept that edit and the test patch then failed (`test_patch_failed`). The check that runs before the patch now only looks up programs and modules that run before an earlier command could set them up (only `cd`, `export` and similar builtins may come first); before, a test command that sets up its own environment, such as `python3 -m venv /tmp/env && /tmp/env/bin/python -m unittest`, was refused (`preflight_failed`) because the venv did not exist yet.
+- **Imported and built-in env rewards.** BigCodeBench checks the tests' top-level imports before the solution runs, so a module the sandbox lacks gives no verdict instead of a 0. The EvalPlus importer no longer publishes a task whose reference fails on any input, and removes an earlier import's copy of it; before, those cases left the suite, so a solution that failed them could score 1.0. `file_organizer` checks each README's description line, not only its heading. `cve_2021_44228_log_triage` scores only the answer's own fields (keywords elsewhere in the output earn nothing), needs the exact CVE ID and `log4shell`, counts payload indicators that name the `jndi:ldap` and `jndi:dns` lookups, and gives no credit for remediation steps in a clause that holds a negation ("do not patch", "patching is not needed").
 
 ## 0.4.2 (2026-10-13)
 

deepgym/scripts/import_bigcodebench.py

@@ -20,6 +20,7 @@
 # Runs inside the isolated solution process: loads the solution, runs the
 # BigCodeBench unittest code against it, prints one JSON result line.
 _RUNNER_SOURCE = """
+import ast
 import importlib.util
 import io
 import json
@@ -37,6 +38,17 @@ def report(score, passed, details, **extra):
         real_stdout.write(json.dumps({"score": score, "passed": passed, "details": details, **extra}) + "\\n")
         real_stdout.flush()
 
+    # The tests and their imports are trusted: check them before the solution runs,
+    # so a module the sandbox lacks is no verdict instead of the solution's 0.
+    # ponytail: only top-level imports are checked; imports inside test methods still
+    # fail as tests, since running other test code needs names the solution defines.
+    try:
+        tree = ast.parse(spec_data["test_code"])
+        imports = [node for node in tree.body if isinstance(node, (ast.Import, ast.ImportFrom))]
+        exec(compile(ast.Module(body=imports, type_ignores=[]), "<tests>", "exec"), {})
+    except Exception as e:
+        return report(0.0, False, f"Test setup error: {type(e).__name__}: {e}", error_type="test_setup_error")
+
     try:
         spec = importlib.util.spec_from_file_location("solution", sys.argv[2])
         mod = importlib.util.module_from_spec(spec)

deepgym/scripts/import_evalplus.py

@@ -10,6 +10,7 @@
 
 import copy
 import json
+import shutil
 import sys
 import traceback
 from pathlib import Path
@@ -46,7 +47,7 @@ def compute_expected_outputs(prompt, canonical_solution, entry_point, inputs):
 
     Returns a list of expected outputs (one per input). If the reference
     solution itself errors on an input, that entry is set to the sentinel
-    string '__REFERENCE_ERROR__' so the verifier can skip it.
+    string '__REFERENCE_ERROR__', and the verifier generator rejects the task.
     """
     ns = {}
     code = prompt + canonical_solution
@@ -82,7 +83,14 @@ def generate_evalplus_verifier(task_id, entry_point, base_inputs, plus_inputs,
 
     For tasks with very large data (huge integers), test data is stored as
     compressed pickle to keep verifier files manageable.
+
+    Raises ValueError when the reference failed on any input: dropping those
+    cases would publish a smaller suite than the benchmark's.
     """
+    errors = sum(1 for e in expected_outputs if e == "__REFERENCE_ERROR__")
+    if errors or not expected_outputs:
+        raise ValueError(f"the reference failed on {errors}/{len(expected_outputs)} inputs")
+
     import pickle as _pickle
     import zlib as _zlib
     import base64 as _base64
@@ -141,8 +149,6 @@ def generate_evalplus_verifier(task_id, entry_point, base_inputs, plus_inputs,
 
 def _compare(result, expected, atol):
     """Compare result to expected, handling floats with tolerance."""
-    if expected == "__REFERENCE_ERROR__":
-        return True  # skip inputs where reference itself errored
     if isinstance(expected, float):
         return isinstance(result, (int, float)) and math.isclose(
             result, expected, abs_tol=atol if atol else 1e-6
@@ -177,13 +183,9 @@ def verify(solution_path):
     base_passed = 0
     plus_passed = 0
     total = len(TEST_INPUTS)
-    skipped = 0
     details = []
 
     for i, (inp, expected) in enumerate(zip(TEST_INPUTS, EXPECTED_OUTPUTS)):
-        if expected == "__REFERENCE_ERROR__":
-            skipped += 1
-            continue
         try:
             result = func(*inp)
             if _compare(result, expected, ATOL):
@@ -202,30 +204,19 @@ def verify(solution_path):
             if len(details) < 5:
                 details.append(f"Test {{i}}: {{type(e).__name__}}: {{e}}")
 
-    effective_total = total - skipped
-    if effective_total == 0:
-        raise RuntimeError("the reference errored on every input, so nothing can be scored")
-    score = passed / effective_total
-    effective_base = NUM_BASE - sum(
-        1 for e in EXPECTED_OUTPUTS[:NUM_BASE] if e == "__REFERENCE_ERROR__"
-    )
-    effective_plus = NUM_PLUS - sum(
-        1 for e in EXPECTED_OUTPUTS[NUM_BASE:] if e == "__REFERENCE_ERROR__"
-    )
-
-    # The score keeps partial credit; passing means every scored test passed.
+    # The score keeps partial credit; passing means every test passed.
     return {{
-        "score": score,
-        "passed": passed == effective_total,
+        "score": passed / total,
+        "passed": passed == total,
         "details": (
-            f"{{passed}}/{{effective_total}} passed "
-            f"(base: {{base_passed}}/{{effective_base}}, "
-            f"plus: {{plus_passed}}/{{effective_plus}}). "
+            f"{{passed}}/{{total}} passed "
+            f"(base: {{base_passed}}/{{NUM_BASE}}, "
+            f"plus: {{plus_passed}}/{{NUM_PLUS}}). "
             + "; ".join(details)
         ),
         "reward_components": {{
-            "base_score": base_passed / effective_base if effective_base > 0 else 0.0,
-            "plus_score": plus_passed / effective_plus if effective_plus > 0 else 0.0,
+            "base_score": base_passed / NUM_BASE if NUM_BASE > 0 else 0.0,
+            "plus_score": plus_passed / NUM_PLUS if NUM_PLUS > 0 else 0.0,
         }}
     }}
 
@@ -269,11 +260,21 @@ def import_benchmark(name, data, output_dir):
             prompt, canonical, entry_point, all_inputs
         )
 
-        ref_errors = sum(1 for o in expected_outputs if o == "__REFERENCE_ERROR__")
-        if ref_errors > 0:
-            errors.append(f"  {safe_id}: {ref_errors}/{len(all_inputs)} reference errors")
-
         env_dir = output_dir / safe_id
+        try:
+            verifier_src = generate_evalplus_verifier(
+                task_id, entry_point, base_inputs, plus_inputs,
+                expected_outputs, atol
+            )
+        except ValueError as e:
+            errors.append(f"  {safe_id}: not imported, {e}")
+            # An earlier import's copy would still load by name.
+            if env_dir.is_symlink():
+                env_dir.unlink()
+            elif env_dir.exists():
+                shutil.rmtree(env_dir)
+            continue
+
         env_dir.mkdir(exist_ok=True)
 
         # task.md
@@ -288,10 +289,6 @@ def import_benchmark(name, data, output_dir):
         (env_dir / "reference_solution.py").write_text(ref_code, encoding="utf-8")
 
         # verifier.py
-        verifier_src = generate_evalplus_verifier(
-            task_id, entry_point, base_inputs, plus_inputs,
-            expected_outputs, atol
-        )
         (env_dir / "verifier.py").write_text(verifier_src, encoding="utf-8")
 
         # metadata.json
@@ -322,7 +319,7 @@ def import_benchmark(name, data, output_dir):
     )
 
     if errors:
-        print(f"  Reference errors in {len(errors)} tasks:")
+        print(f"  Skipped {len(errors)} tasks with reference errors:")
         for e in errors[:10]:
             print(e)
         if len(errors) > 10: