fix: reject unknown attacks and invalid sub-scores, drop stale QA results, keep export runtimes and initial observations

deepgym/CHANGELOG.md

@@ -23,6 +23,7 @@
 - **Core reward path: each failure lands on whoever caused it.** A verifier that prints a score and then crashes no longer keeps that score, in any verifier shape; before, Python's exit 1 for an uncaught exception read as the protocol's "failed" and the printed partial score reached the reward function. An exception hook now decides, while the script keeps running as `__main__` with its own `__file__`: before the policy's output is in play, the crash is the verifier's own failure, with no verdict (`runtime_error`, returned rather than raised); once it is (a solution process started, or the verifier opened the solution's file, or a session verifier opened a file in the agent's work directory, so running the solution with `exec` and parsing malformed policy output count), the crash came from handling it, so it is a judged 0 with `error_type='solution_error'`, a new model fault. So is a result the verifier can't normalize (NaN, not a number) after that point. Whatever the solution does (raise, exit, hang, die, return garbage, print unparseable output) is therefore its own 0 even when the verifier doesn't handle it; before, an uncaught solution error became a verifier error with no verdict, as `import_error` or `timeout` when the solution raised `ImportError` or `TimeoutError` itself, so a solution could dodge its score by breaking the verifier. Every failure from the solution process now reaches the verifier as a `deepgym_solution.SolutionError`, still of the builtin type it raised (`except ValueError` and `except TimeoutError` keep working). A solution that runs past its time limit is a judged 0 with `error_type='solution_timeout'`, a new model fault, on every path; `timeout` stays DeepGym's own timeout, with no verdict. Load test data before the solution: a verifier that fails after starting it scores 0. `LaunchError` is now a `BaseException`, so a verifier's `except Exception` can't score the machine's failure; uncaught, it ends the run with `sandbox_error` (no verdict). Before the first solution start, a preflight checks free disk and inodes and that the isolation launcher (sudo, setpriv, bwrap, sandbox-exec) starts code from a fresh directory, and a judge-side failure while preparing the run (a temp file, the loopback relay) is a `LaunchError` too. Once solution code has run, a failed relaunch is the solution's, a `SolutionError` scored 0, since it may have used up the disk or processes. The judge prints a per-run token on stdout once the policy's output is in play. A single-turn sandbox that dies after that token (its Docker or Apple container stops, or a Modal exec ends with the sandbox finished, which return what was printed) is the policy's doing: a judged 0 with `error_type='policy_sandbox_failure'`, where it was `sandbox_error`; so is a run that outruns its time limit after the token, locally or in a sandbox (`solution_timeout`), such as a verifier regex the solution's output made backtrack. One that dies before the token (creation, setup, uploads, the verifier loading its own fixtures), one whose provider returns nothing printed (Daytona, an E2B or Modal call that raises), or one Daytona evicts or times out in its queue (as its error or its refreshed sandbox record says) keeps no verdict. Wrappers report it as the new `deepgym.exceptions.SandboxDiedError` (also in `deepgym.sandboxes`), which `is_infrastructure_failure` never counts as the provider's failure. Every verdict line the harness prints starts a fresh stdout line, so a partial line the verifier left unterminated can't hide it, and a stderr the solution broke (a full disk under a file-backed stderr) can't change the verdict or the exit code. Session verifiers use the same guard, and the code they load gets the step's time limit as its deadline, as the judge does, with a few seconds' grace for the verifier to report. A session step that fails after its action may have run (the provider failing while it applies the action, takes the screenshot or runs the step verifier, not only a `VerifierError`) now ends the episode: the action stays on the trajectory with no step reward, further steps are refused, and `score()` returns no verdict for a network or provider failure (`is_infrastructure_failure`) or a failed step verifier; anything else follows from the agent's actions (its sandbox died, a write failed, its files, a folder or a planted link, blocked the step-output file) and is a judged 0 (`error_type='policy_sandbox_failure'`). A sandbox that dies under the final verifier is a judged 0 too, once the agent has acted. `run_episode` returns the result a failed step fixed instead of raising. Before, the session stayed runnable, so a retry applied a second action past `max_steps`, scored it, and the first action was never recorded. The new observation is now taken before the step verifier runs, so a failed screenshot no longer leaves a recorded reward and a stale observation behind for a retry to replay, and trace replay (`Trace.to_trajectory()`) keeps a recorded step that failed without an observation; trace capture also keeps a step interrupted after its action ran. A `write_file` blocked by the agent's own files (a folder or link in the way, found before anything is written) raises `ActionError` (rejected, the episode goes on); any other write failure is a `SandboxError`. The judge's setup refuses an existing `dgsolver` account that is root or shares its uid, gid or groups with another account (the check session setup already ran, now shared and also refusing uid or gid 0), and the judge never drops the solution to uid or gid 0; before, an image with a root-alias `dgsolver` ran the solution as root. Nothing a solution starts outlives its verifier any more: with its first solution the verifier starts a watcher process, in its own session, that kills every solution process group (a `load_solution` server, a `run_solution` script and whatever they started) once the verifier ends, however it ends (a host timeout or cancel, a crash), even when the solution hogs the interpreter in a long C call; a `run_solution` script waits to start until the watcher knows it, and a verifier's own exit waits for the watcher, so a sandbox user's sweep never reaches the next grade (on macOS, a child that detaches into a session of its own still escapes, which is why subprocesses need `allow_subprocess` there); before, such a server, or a busy child a solution started, kept running after the run. A cancelled or timed-out local run, sync or async, kills its verifier's whole process group and reaps the verifier, so neither a verifier's child nor one holding its output pipes outlives or stalls it, and a traced session's `close()` retries a sandbox delete that failed instead of returning silently.
 - **SWE-bench Pro / MiMo patch runner: two cases where a correct patch got no grade.** In a task whose `repo_dir` is a subdirectory of the repository, the hidden test files were saved and restored at the wrong path, so a patch that also edited a hidden test kept that edit and the test patch then failed (`test_patch_failed`). The check that runs before the patch now only looks up programs and modules that run before an earlier command could set them up (only `cd`, `export` and similar builtins may come first); before, a test command that sets up its own environment, such as `python3 -m venv /tmp/env && /tmp/env/bin/python -m unittest`, was refused (`preflight_failed`) because the venv did not exist yet.
 - **Imported and built-in env rewards.** BigCodeBench checks the tests' top-level imports before the solution runs, so a module the sandbox lacks gives no verdict instead of a 0. The EvalPlus importer no longer publishes a task whose reference fails on any input, and removes an earlier import's copy of it; before, those cases left the suite, so a solution that failed them could score 1.0. `file_organizer` checks each README's description line, not only its heading. `cve_2021_44228_log_triage` scores only the answer's own fields (keywords elsewhere in the output earn nothing), needs the exact CVE ID and `log4shell`, counts payload indicators that name the `jndi:ldap` and `jndi:dns` lookups, and gives no credit for remediation steps in a clause that holds a negation ("do not patch", "patching is not needed").
+- **QA evidence, sub-scores and exports.** An unknown attack strategy is rejected before any attack runs (a 422 over the API); a misspelt one ran nothing and read as robust. An inconclusive retest of a changed verifier or attack drops its old stored result instead of leaving it to stand in. A NaN or infinite reward component, or an invalid case score, is no longer a trainable 0 under a valid overall score: the verifier wrapper checks them as it does the overall score, so one the solution's output made invalid is its judged 0 (`solution_error`), and one the verifier produced on its own raises `InvalidScoreError` (no verdict). OpenRLHF's router rewards a sample whose verifier failed or returned an invalid result its `infra_failure_score` (NaN in `'nan'` mode) instead of 0; the model's own failures, and a run still going at the request deadline, still score 0. Harbor export refuses snapshot-backed environments instead of exporting them on a plain image, and RL trace rows carry `initial_observation` and `initial_screenshot`.
 
 ## 0.4.2 (2026-10-13)
 

deepgym/README.md

@@ -207,9 +207,9 @@ app.include_router(create_openrlhf_router(
 # OpenRLHF: --remote_rm_url http://localhost:8000/reward/score --algo.dynamic_filtering_enable
 ```
 
-OpenRLHF can't skip a sample, so `infra_failure_score` decides what a sample gets when the
-provider or the network failed it: `'nan'` (dynamic filtering then drops its group) or
-`'zero'`. Other runs without a verdict score 0. More in [OpenRLHF][openrlhf].
+OpenRLHF can't skip a sample, so `infra_failure_score` decides what a sample DeepGym
+couldn't grade gets: `'nan'` (dynamic filtering then drops its group) or `'zero'`. More in
+[OpenRLHF][openrlhf].
 
 ### Evaluate with lm-evaluation-harness
 

deepgym/docs/training/openrlhf.md

@@ -35,21 +35,21 @@ would stop its training step, so the endpoint never fails a well-formed request
 run. A malformed or oversized body gets 422, and `prompts` is required even when the
 client sends `outputs`.
 
-- A failure the model caused is judged and scores 0, and so does any run without a
-  verdict that the model's output may have caused (a verifier failure, a timeout); those
-  are never run again.
+- A failure the model caused (a solution that crashed, hung or broke its sandbox) is
+  judged and scores 0.
 - A run the sandbox provider or the network failed
   (`deepgym.exceptions.is_infrastructure_failure`) is graded once more after a short
-  pause.
+  pause. Any other run without a verdict (a verifier that failed before judging, say) is
+  never run again.
 - Every run, first or second, takes one of the router's `max_parallel` slots (32 by
   default), and no request waits past 150 s (OpenRLHF gives up at 180 s). A first run
   still going then scores 0, since the model's code may be what slows it.
 
 ## Samples DeepGym can't grade
 
-If the second grading fails too or doesn't finish in time, the first one never got a slot
-in time, or a query doesn't contain its prompt, the sample gets the router's
-`infra_failure_score`, never a score from a run that didn't finish. It has no default:
+A sample left without a verdict (its grading failed, a second grading failed too or
+didn't finish in time, it never got a slot in time, or its query doesn't contain its
+prompt) gets the router's `infra_failure_score`, never a score from a run that didn't finish. It has no default:
 creating the router without it raises an error, so the choice is made at setup, where no
 policy can reach it.
 
@@ -61,6 +61,5 @@ policy can reach it.
   leak.
 
 Either way the sample is logged as a warning with its error type, and counts toward a
-warning logged when more than 5 samples in a block of 100 get `infra_failure_score`. Runs
-without a verdict that score 0 (above) aren't counted. See
+warning logged when more than 5 samples in a block of 100 get `infra_failure_score`. See
 [the reward contract](reward-contract.md).

deepgym/docs/training/reward-contract.md

@@ -67,7 +67,7 @@ use this test.
 | verl `compute_score` | Raises `NoVerdictError`. verl's v1 trainer drops the sample; the legacy trainer stops, so use the `deepgym_single_turn` agent loop there. |
 | verl agent loops (`deepgym_single_turn`, `deepgym_agent`) | Dropped under verl's v1 trainer, masked under the legacy trainer, logged as `error` 1.0. |
 | verl `make_verl_reward_fn` (custom loops) | `None`. |
-| OpenRLHF endpoint | Graded once more if the provider or the network failed it, then the router's `infra_failure_score`. Other runs without a verdict score 0. |
+| OpenRLHF endpoint | Graded once more if the provider or the network failed it; any sample still without a verdict gets the router's `infra_failure_score`. |
 | lm-eval task | Left out of `deepgym_score` and `deepgym_pass`; `deepgym_verdict` is the share that got one. |
 | `DeepGymEnv` | `info.has_verdict` is false; mask the sample. `step` raises where `dg.run` does; `step_batch` returns those runs without a verdict. |
 

deepgym/src/deepgym/adversarial.py

@@ -22,7 +22,7 @@
 import os
 import re
 import textwrap
-from typing import TYPE_CHECKING, Literal
+from typing import TYPE_CHECKING, Literal, get_args
 
 from pydantic import BaseModel, Field
 
@@ -38,7 +38,7 @@
 # Default attack strategies
 # ---------------------------------------------------------------------------
 
-DEFAULT_STRATEGIES: list[str] = [
+AttackStrategy = Literal[
     'empty',
     'hardcoded',
     'trivial',
@@ -47,6 +47,7 @@
     'patch_tampering',
     'llm_attack',
 ]
+DEFAULT_STRATEGIES: list[str] = list(get_args(AttackStrategy))
 
 PASS_THRESHOLD: float = 0.5  # score above this ⇒ exploit found
 
@@ -361,8 +362,16 @@ def test(
         Returns
         -------
         AdversarialReport
+
+        Raises
+        ------
+        ValueError
+            If a strategy isn't one of :data:`DEFAULT_STRATEGIES`; nothing runs.
         """
         strategies = strategies or list(DEFAULT_STRATEGIES)
+        # A misspelt strategy would run nothing and still read as robust.
+        if unknown := sorted(set(strategies) - set(DEFAULT_STRATEGIES)):
+            raise ValueError(f'Unknown attack strategies {unknown}; known: {DEFAULT_STRATEGIES}')
         results: list[AttackResult] = []
 
         for strategy in strategies:
@@ -417,7 +426,7 @@ def _generate_attack(self, env: Environment, strategy: str) -> str:
         elif strategy == 'llm_attack':
             return self._attack_llm(env)
         else:
-            return f'# Unknown strategy: {strategy}\npass\n'
+            raise ValueError(f'Unknown attack strategy {strategy!r}')
         body = _ENTRY_POINT_BODIES[strategy]
         return code + ''.join(
             f'\n\ndef {name}(*args, **kwargs):\n    {body}\n' for name in entry_points(env)

deepgym/src/deepgym/api/schemas.py

@@ -7,6 +7,7 @@
 from pydantic import BaseModel, Field, field_validator, model_validator
 
 from deepgym import __version__
+from deepgym.adversarial import AttackStrategy
 from deepgym.benchmark_ops import BenchmarkAuditReport
 from deepgym.computer_use import ComputerUseEnvironment
 from deepgym.models import Environment, MultiTurnEnvironment, Observation, VerifierMode
@@ -168,7 +169,7 @@ class VerifierAuditRequest(BaseModel):
     environment: APIEnvironment
     verifier_id: str | None = Field(default=None, max_length=200)
     benchmark: str = Field(default='custom', max_length=200)
-    strategies: list[str] | None = Field(default=None, max_length=20)
+    strategies: list[AttackStrategy] | None = Field(default=None, max_length=20)
     persist: bool = False
 
 
@@ -177,7 +178,7 @@ class RegisteredVerifierAuditRequest(BaseModel):
 
     verifier_id: str | None = Field(default=None, max_length=200)
     benchmark: str = Field(default='registered', max_length=200)
-    strategies: list[str] | None = Field(default=None, max_length=20)
+    strategies: list[AttackStrategy] | None = Field(default=None, max_length=20)
     persist: bool = False
 
 

deepgym/src/deepgym/harbor/export.py

@@ -84,16 +84,22 @@ def export_environment(env: Environment | MultiTurnEnvironment, out_dir: Path) -
         The new task folder.
 
     Raises:
-        DeepGymError: If *env* is multi-turn, unnamed, benchmark-backed or a Harbor task
-            already, if the folder exists, or if its reference solution contains the
-            heredoc marker.
+        DeepGymError: If *env* is multi-turn, unnamed, benchmark-backed, snapshot-backed
+            or a Harbor task already, if the folder exists, or if its reference solution
+            contains the heredoc marker.
     """
     if not isinstance(env, Environment):
         raise DeepGymError(f'{env.name} is multi-turn; only single-turn envs export')
     if hasattr(env, 'run_with_deepgym') or hasattr(env, 'prepare_batch_requests'):
         raise DeepGymError(f'{env.name} scores itself, so it has no verifier to export')
     if not env.name or PurePosixPath(env.name).name != env.name or env.name in ('.', '..'):
         raise DeepGymError(f'Cannot name a task folder {env.name!r}')
+    if env.snapshot:
+        raise DeepGymError(
+            f'{env.name} boots from the snapshot {env.snapshot!r}, which a Harbor task '
+            "can't carry; to export it, drop the snapshot and set an image that has its "
+            'dependencies'
+        )
     task_dir = out_dir / env.name
     if task_dir.exists():
         raise DeepGymError(f'{task_dir} already exists; remove it to export again')