diff --git a/deepgym/CHANGELOG.md b/deepgym/CHANGELOG.md index 0843e53..39f016a 100644 --- a/deepgym/CHANGELOG.md +++ b/deepgym/CHANGELOG.md @@ -23,6 +23,7 @@ - **Core reward path: each failure lands on whoever caused it.** A verifier that prints a score and then crashes no longer keeps that score, in any verifier shape; before, Python's exit 1 for an uncaught exception read as the protocol's "failed" and the printed partial score reached the reward function. An exception hook now decides, while the script keeps running as `__main__` with its own `__file__`: before the policy's output is in play, the crash is the verifier's own failure, with no verdict (`runtime_error`, returned rather than raised); once it is (a solution process started, or the verifier opened the solution's file, or a session verifier opened a file in the agent's work directory, so running the solution with `exec` and parsing malformed policy output count), the crash came from handling it, so it is a judged 0 with `error_type='solution_error'`, a new model fault. So is a result the verifier can't normalize (NaN, not a number) after that point. Whatever the solution does (raise, exit, hang, die, return garbage, print unparseable output) is therefore its own 0 even when the verifier doesn't handle it; before, an uncaught solution error became a verifier error with no verdict, as `import_error` or `timeout` when the solution raised `ImportError` or `TimeoutError` itself, so a solution could dodge its score by breaking the verifier. Every failure from the solution process now reaches the verifier as a `deepgym_solution.SolutionError`, still of the builtin type it raised (`except ValueError` and `except TimeoutError` keep working). A solution that runs past its time limit is a judged 0 with `error_type='solution_timeout'`, a new model fault, on every path; `timeout` stays DeepGym's own timeout, with no verdict. Load test data before the solution: a verifier that fails after starting it scores 0. `LaunchError` is now a `BaseException`, so a verifier's `except Exception` can't score the machine's failure; uncaught, it ends the run with `sandbox_error` (no verdict). Before the first solution start, a preflight checks free disk and inodes and that the isolation launcher (sudo, setpriv, bwrap, sandbox-exec) starts code from a fresh directory, and a judge-side failure while preparing the run (a temp file, the loopback relay) is a `LaunchError` too. Once solution code has run, a failed relaunch is the solution's, a `SolutionError` scored 0, since it may have used up the disk or processes. The judge prints a per-run token on stdout once the policy's output is in play. A single-turn sandbox that dies after that token (its Docker or Apple container stops, or a Modal exec ends with the sandbox finished, which return what was printed) is the policy's doing: a judged 0 with `error_type='policy_sandbox_failure'`, where it was `sandbox_error`; so is a run that outruns its time limit after the token, locally or in a sandbox (`solution_timeout`), such as a verifier regex the solution's output made backtrack. One that dies before the token (creation, setup, uploads, the verifier loading its own fixtures), one whose provider returns nothing printed (Daytona, an E2B or Modal call that raises), or one Daytona evicts or times out in its queue (as its error or its refreshed sandbox record says) keeps no verdict. Wrappers report it as the new `deepgym.exceptions.SandboxDiedError` (also in `deepgym.sandboxes`), which `is_infrastructure_failure` never counts as the provider's failure. Every verdict line the harness prints starts a fresh stdout line, so a partial line the verifier left unterminated can't hide it, and a stderr the solution broke (a full disk under a file-backed stderr) can't change the verdict or the exit code. Session verifiers use the same guard, and the code they load gets the step's time limit as its deadline, as the judge does, with a few seconds' grace for the verifier to report. A session step that fails after its action may have run (the provider failing while it applies the action, takes the screenshot or runs the step verifier, not only a `VerifierError`) now ends the episode: the action stays on the trajectory with no step reward, further steps are refused, and `score()` returns no verdict for a network or provider failure (`is_infrastructure_failure`) or a failed step verifier; anything else follows from the agent's actions (its sandbox died, a write failed, its files, a folder or a planted link, blocked the step-output file) and is a judged 0 (`error_type='policy_sandbox_failure'`). A sandbox that dies under the final verifier is a judged 0 too, once the agent has acted. `run_episode` returns the result a failed step fixed instead of raising. Before, the session stayed runnable, so a retry applied a second action past `max_steps`, scored it, and the first action was never recorded. The new observation is now taken before the step verifier runs, so a failed screenshot no longer leaves a recorded reward and a stale observation behind for a retry to replay, and trace replay (`Trace.to_trajectory()`) keeps a recorded step that failed without an observation; trace capture also keeps a step interrupted after its action ran. A `write_file` blocked by the agent's own files (a folder or link in the way, found before anything is written) raises `ActionError` (rejected, the episode goes on); any other write failure is a `SandboxError`. The judge's setup refuses an existing `dgsolver` account that is root or shares its uid, gid or groups with another account (the check session setup already ran, now shared and also refusing uid or gid 0), and the judge never drops the solution to uid or gid 0; before, an image with a root-alias `dgsolver` ran the solution as root. Nothing a solution starts outlives its verifier any more: with its first solution the verifier starts a watcher process, in its own session, that kills every solution process group (a `load_solution` server, a `run_solution` script and whatever they started) once the verifier ends, however it ends (a host timeout or cancel, a crash), even when the solution hogs the interpreter in a long C call; a `run_solution` script waits to start until the watcher knows it, and a verifier's own exit waits for the watcher, so a sandbox user's sweep never reaches the next grade (on macOS, a child that detaches into a session of its own still escapes, which is why subprocesses need `allow_subprocess` there); before, such a server, or a busy child a solution started, kept running after the run. A cancelled or timed-out local run, sync or async, kills its verifier's whole process group and reaps the verifier, so neither a verifier's child nor one holding its output pipes outlives or stalls it, and a traced session's `close()` retries a sandbox delete that failed instead of returning silently. - **SWE-bench Pro / MiMo patch runner: two cases where a correct patch got no grade.** In a task whose `repo_dir` is a subdirectory of the repository, the hidden test files were saved and restored at the wrong path, so a patch that also edited a hidden test kept that edit and the test patch then failed (`test_patch_failed`). The check that runs before the patch now only looks up programs and modules that run before an earlier command could set them up (only `cd`, `export` and similar builtins may come first); before, a test command that sets up its own environment, such as `python3 -m venv /tmp/env && /tmp/env/bin/python -m unittest`, was refused (`preflight_failed`) because the venv did not exist yet. - **Imported and built-in env rewards.** BigCodeBench checks the tests' top-level imports before the solution runs, so a module the sandbox lacks gives no verdict instead of a 0. The EvalPlus importer no longer publishes a task whose reference fails on any input, and removes an earlier import's copy of it; before, those cases left the suite, so a solution that failed them could score 1.0. `file_organizer` checks each README's description line, not only its heading. `cve_2021_44228_log_triage` scores only the answer's own fields (keywords elsewhere in the output earn nothing), needs the exact CVE ID and `log4shell`, counts payload indicators that name the `jndi:ldap` and `jndi:dns` lookups, and gives no credit for remediation steps in a clause that holds a negation ("do not patch", "patching is not needed"). +- **QA evidence, sub-scores and exports.** An unknown attack strategy is rejected before any attack runs (a 422 over the API); a misspelt one ran nothing and read as robust. An inconclusive retest of a changed verifier or attack drops its old stored result instead of leaving it to stand in. A NaN or infinite reward component, or an invalid case score, is no longer a trainable 0 under a valid overall score: the verifier wrapper checks them as it does the overall score, so one the solution's output made invalid is its judged 0 (`solution_error`), and one the verifier produced on its own raises `InvalidScoreError` (no verdict). OpenRLHF's router rewards a sample whose verifier failed or returned an invalid result its `infra_failure_score` (NaN in `'nan'` mode) instead of 0; the model's own failures, and a run still going at the request deadline, still score 0. Harbor export refuses snapshot-backed environments instead of exporting them on a plain image, and RL trace rows carry `initial_observation` and `initial_screenshot`. ## 0.4.2 (2026-10-13) diff --git a/deepgym/README.md b/deepgym/README.md index fddafed..527c59b 100644 --- a/deepgym/README.md +++ b/deepgym/README.md @@ -207,9 +207,9 @@ app.include_router(create_openrlhf_router( # OpenRLHF: --remote_rm_url http://localhost:8000/reward/score --algo.dynamic_filtering_enable ``` -OpenRLHF can't skip a sample, so `infra_failure_score` decides what a sample gets when the -provider or the network failed it: `'nan'` (dynamic filtering then drops its group) or -`'zero'`. Other runs without a verdict score 0. More in [OpenRLHF][openrlhf]. +OpenRLHF can't skip a sample, so `infra_failure_score` decides what a sample DeepGym +couldn't grade gets: `'nan'` (dynamic filtering then drops its group) or `'zero'`. More in +[OpenRLHF][openrlhf]. ### Evaluate with lm-evaluation-harness diff --git a/deepgym/docs/training/openrlhf.md b/deepgym/docs/training/openrlhf.md index e52dcce..c6629f1 100644 --- a/deepgym/docs/training/openrlhf.md +++ b/deepgym/docs/training/openrlhf.md @@ -35,21 +35,21 @@ would stop its training step, so the endpoint never fails a well-formed request run. A malformed or oversized body gets 422, and `prompts` is required even when the client sends `outputs`. -- A failure the model caused is judged and scores 0, and so does any run without a - verdict that the model's output may have caused (a verifier failure, a timeout); those - are never run again. +- A failure the model caused (a solution that crashed, hung or broke its sandbox) is + judged and scores 0. - A run the sandbox provider or the network failed (`deepgym.exceptions.is_infrastructure_failure`) is graded once more after a short - pause. + pause. Any other run without a verdict (a verifier that failed before judging, say) is + never run again. - Every run, first or second, takes one of the router's `max_parallel` slots (32 by default), and no request waits past 150 s (OpenRLHF gives up at 180 s). A first run still going then scores 0, since the model's code may be what slows it. ## Samples DeepGym can't grade -If the second grading fails too or doesn't finish in time, the first one never got a slot -in time, or a query doesn't contain its prompt, the sample gets the router's -`infra_failure_score`, never a score from a run that didn't finish. It has no default: +A sample left without a verdict (its grading failed, a second grading failed too or +didn't finish in time, it never got a slot in time, or its query doesn't contain its +prompt) gets the router's `infra_failure_score`, never a score from a run that didn't finish. It has no default: creating the router without it raises an error, so the choice is made at setup, where no policy can reach it. @@ -61,6 +61,5 @@ policy can reach it. leak. Either way the sample is logged as a warning with its error type, and counts toward a -warning logged when more than 5 samples in a block of 100 get `infra_failure_score`. Runs -without a verdict that score 0 (above) aren't counted. See +warning logged when more than 5 samples in a block of 100 get `infra_failure_score`. See [the reward contract](reward-contract.md). diff --git a/deepgym/docs/training/reward-contract.md b/deepgym/docs/training/reward-contract.md index 6d7bdcd..8bc2a0c 100644 --- a/deepgym/docs/training/reward-contract.md +++ b/deepgym/docs/training/reward-contract.md @@ -67,7 +67,7 @@ use this test. | verl `compute_score` | Raises `NoVerdictError`. verl's v1 trainer drops the sample; the legacy trainer stops, so use the `deepgym_single_turn` agent loop there. | | verl agent loops (`deepgym_single_turn`, `deepgym_agent`) | Dropped under verl's v1 trainer, masked under the legacy trainer, logged as `error` 1.0. | | verl `make_verl_reward_fn` (custom loops) | `None`. | -| OpenRLHF endpoint | Graded once more if the provider or the network failed it, then the router's `infra_failure_score`. Other runs without a verdict score 0. | +| OpenRLHF endpoint | Graded once more if the provider or the network failed it; any sample still without a verdict gets the router's `infra_failure_score`. | | lm-eval task | Left out of `deepgym_score` and `deepgym_pass`; `deepgym_verdict` is the share that got one. | | `DeepGymEnv` | `info.has_verdict` is false; mask the sample. `step` raises where `dg.run` does; `step_batch` returns those runs without a verdict. | diff --git a/deepgym/src/deepgym/adversarial.py b/deepgym/src/deepgym/adversarial.py index 86ec8f3..6dc31ad 100644 --- a/deepgym/src/deepgym/adversarial.py +++ b/deepgym/src/deepgym/adversarial.py @@ -22,7 +22,7 @@ import os import re import textwrap -from typing import TYPE_CHECKING, Literal +from typing import TYPE_CHECKING, Literal, get_args from pydantic import BaseModel, Field @@ -38,7 +38,7 @@ # Default attack strategies # --------------------------------------------------------------------------- -DEFAULT_STRATEGIES: list[str] = [ +AttackStrategy = Literal[ 'empty', 'hardcoded', 'trivial', @@ -47,6 +47,7 @@ 'patch_tampering', 'llm_attack', ] +DEFAULT_STRATEGIES: list[str] = list(get_args(AttackStrategy)) PASS_THRESHOLD: float = 0.5 # score above this ⇒ exploit found @@ -361,8 +362,16 @@ def test( Returns ------- AdversarialReport + + Raises + ------ + ValueError + If a strategy isn't one of :data:`DEFAULT_STRATEGIES`; nothing runs. """ strategies = strategies or list(DEFAULT_STRATEGIES) + # A misspelt strategy would run nothing and still read as robust. + if unknown := sorted(set(strategies) - set(DEFAULT_STRATEGIES)): + raise ValueError(f'Unknown attack strategies {unknown}; known: {DEFAULT_STRATEGIES}') results: list[AttackResult] = [] for strategy in strategies: @@ -417,7 +426,7 @@ def _generate_attack(self, env: Environment, strategy: str) -> str: elif strategy == 'llm_attack': return self._attack_llm(env) else: - return f'# Unknown strategy: {strategy}\npass\n' + raise ValueError(f'Unknown attack strategy {strategy!r}') body = _ENTRY_POINT_BODIES[strategy] return code + ''.join( f'\n\ndef {name}(*args, **kwargs):\n {body}\n' for name in entry_points(env) diff --git a/deepgym/src/deepgym/api/schemas.py b/deepgym/src/deepgym/api/schemas.py index f357593..a5f5554 100644 --- a/deepgym/src/deepgym/api/schemas.py +++ b/deepgym/src/deepgym/api/schemas.py @@ -7,6 +7,7 @@ from pydantic import BaseModel, Field, field_validator, model_validator from deepgym import __version__ +from deepgym.adversarial import AttackStrategy from deepgym.benchmark_ops import BenchmarkAuditReport from deepgym.computer_use import ComputerUseEnvironment from deepgym.models import Environment, MultiTurnEnvironment, Observation, VerifierMode @@ -168,7 +169,7 @@ class VerifierAuditRequest(BaseModel): environment: APIEnvironment verifier_id: str | None = Field(default=None, max_length=200) benchmark: str = Field(default='custom', max_length=200) - strategies: list[str] | None = Field(default=None, max_length=20) + strategies: list[AttackStrategy] | None = Field(default=None, max_length=20) persist: bool = False @@ -177,7 +178,7 @@ class RegisteredVerifierAuditRequest(BaseModel): verifier_id: str | None = Field(default=None, max_length=200) benchmark: str = Field(default='registered', max_length=200) - strategies: list[str] | None = Field(default=None, max_length=20) + strategies: list[AttackStrategy] | None = Field(default=None, max_length=20) persist: bool = False diff --git a/deepgym/src/deepgym/harbor/export.py b/deepgym/src/deepgym/harbor/export.py index 5141967..bd9ab48 100644 --- a/deepgym/src/deepgym/harbor/export.py +++ b/deepgym/src/deepgym/harbor/export.py @@ -84,9 +84,9 @@ def export_environment(env: Environment | MultiTurnEnvironment, out_dir: Path) - The new task folder. Raises: - DeepGymError: If *env* is multi-turn, unnamed, benchmark-backed or a Harbor task - already, if the folder exists, or if its reference solution contains the - heredoc marker. + DeepGymError: If *env* is multi-turn, unnamed, benchmark-backed, snapshot-backed + or a Harbor task already, if the folder exists, or if its reference solution + contains the heredoc marker. """ if not isinstance(env, Environment): raise DeepGymError(f'{env.name} is multi-turn; only single-turn envs export') @@ -94,6 +94,12 @@ def export_environment(env: Environment | MultiTurnEnvironment, out_dir: Path) - raise DeepGymError(f'{env.name} scores itself, so it has no verifier to export') if not env.name or PurePosixPath(env.name).name != env.name or env.name in ('.', '..'): raise DeepGymError(f'Cannot name a task folder {env.name!r}') + if env.snapshot: + raise DeepGymError( + f'{env.name} boots from the snapshot {env.snapshot!r}, which a Harbor task ' + "can't carry; to export it, drop the snapshot and set an image that has its " + 'dependencies' + ) task_dir = out_dir / env.name if task_dir.exists(): raise DeepGymError(f'{task_dir} already exists; remove it to export again') diff --git a/deepgym/src/deepgym/integrations/openrlhf.py b/deepgym/src/deepgym/integrations/openrlhf.py index b65fd18..b1ea2be 100644 --- a/deepgym/src/deepgym/integrations/openrlhf.py +++ b/deepgym/src/deepgym/integrations/openrlhf.py @@ -29,18 +29,19 @@ prompt's whole group when the mean of its scores falls outside ``--algo.dynamic_filtering_range``; a NaN always does. -So the endpoint never fails a request over a run. A judged run gets its score. A run -without a verdict for a reason the model may have caused (the verifier failed, the run -timed out, the sandbox broke) scores 0 in either mode, with a warning, and is never run -again. A run the sandbox provider or the network failed (see +So the endpoint never fails a request over a run. A judged run gets its score, and the +model's own failures (a solution that crashed, hung or broke its sandbox) are judged 0s. +A run the sandbox provider or the network failed (see :func:`~deepgym.exceptions.is_infrastructure_failure`) is graded once more after a short -pause. Every run, first or repeated, takes one of the router's ``max_parallel`` slots, -and no request waits past 150 s, inside the 180 s OpenRLHF gives it; a first run still -going then scores 0, since the model's code may be what slows it. If the second grading -fails too, can't start in time or is still going then, or the first one never started, -or a query doesn't contain its prompt, the sample is rewarded the router's -``infra_failure_score``, never a score from a run that didn't finish. The router has no -default for it, so the choice is made when it's set up, where no policy can reach it: +pause; any other run without a verdict (a verifier that failed before judging, say) is +never run again. Every run, first or repeated, takes one of the router's +``max_parallel`` slots, and no request waits past 150 s, inside the 180 s OpenRLHF gives +it; a first run still going then scores 0, since the model's code may be what slows it +(a NaN would let slow code dodge its 0). A sample left without a verdict otherwise, +because its grading failed or never started, or because its query doesn't contain its +prompt, is rewarded the router's ``infra_failure_score``, never a score from a run that +didn't finish. The router has no default for it, so the choice is made when it's set +up, where no policy can reach it: - ``'nan'`` (recommended): NaN, so dynamic filtering drops the sample's group. It needs ``--algo.dynamic_filtering_enable``: without it OpenRLHF trains on the NaN and the run @@ -190,9 +191,9 @@ def _grade(dg: DeepGym, env: Environment, code: str, deadline: float) -> RunResu A run the provider or the network failed (see :func:`~deepgym.exceptions.is_infrastructure_failure`), or one whose slot came after *deadline* (``time.monotonic``), which then doesn't start, comes back as its error_type: - there is no score to give. Anything else the model may have caused, so it comes back as - a result without a verdict, scored 0: a run that raised for any other reason, or one - that ended after *deadline* (the model's code may be what slowed it). + there is no score to give, but a second grading may give one. A run that raised for any + other reason comes back as a result without a verdict, and is never graded again; one + that ended after *deadline* comes back as :func:`_unfinished`. """ if time.monotonic() >= deadline: return DEADLINE_EXCEEDED @@ -255,8 +256,8 @@ def create_openrlhf_router( Args: env: The environment every response is scored with. dg: The DeepGym client that scores them. - infra_failure_score: Required. The reward of a sample DeepGym couldn't grade - through no fault of the model: ``'nan'`` (recommended, with OpenRLHF's dynamic + infra_failure_score: Required. The reward of a sample DeepGym couldn't grade, + one without a verdict: ``'nan'`` (recommended, with OpenRLHF's dynamic filtering) or ``'zero'`` (see the module docstring). max_parallel: Most runs going at once across every request, first gradings and second ones alike. @@ -266,8 +267,8 @@ def create_openrlhf_router( """ if infra_failure_score not in ('nan', 'zero'): raise ValueError( - "Choose infra_failure_score, the reward of a sample DeepGym couldn't grade because " - "the sandbox provider or the network failed (OpenRLHF can't skip a sample): 'nan' " + "Choose infra_failure_score, the reward of a sample DeepGym couldn't grade, one " + "without a verdict (OpenRLHF can't skip a sample): 'nan' " '(recommended) drops its prompt group, and needs --algo.dynamic_filtering_enable, ' "without which the NaN breaks training; 'zero' trains on it as a failed answer, " 'an accepted leak that is logged and counted toward the masked-rate warning.' @@ -297,19 +298,16 @@ async def score(request: OpenRLHFRewardRequest) -> Response: error_types: list[str | None] = [] for i in range(len(responses)): outcome = outcomes.get(i, PROMPT_MISMATCH) - masked.record(masked=isinstance(outcome, str)) - if isinstance(outcome, str): - logger.warning('OpenRLHF: no verdict (%s); rewarded %s', outcome, fallback) - rewards.append(fallback) - error_types.append(outcome) - continue - if not outcome.has_verdict: - logger.warning( - 'OpenRLHF: no verdict (%s), maybe the model output; rewarded 0', - outcome.error_type, - ) - rewards.append(outcome.score if outcome.has_verdict else 0.0) - error_types.append(outcome.error_type) + error_type = outcome if isinstance(outcome, str) else outcome.error_type + judged = isinstance(outcome, RunResult) and outcome.has_verdict + # A run still going at the deadline scores 0: the model's code may be the cause. + unfinished = isinstance(outcome, RunResult) and error_type == DEADLINE_EXCEEDED + masked.record(masked=not (judged or unfinished)) + reward = outcome.score if judged else 0.0 if unfinished else fallback + if not judged: + logger.warning('OpenRLHF: no verdict (%s); rewarded %s', error_type, reward) + rewards.append(reward) + error_types.append(error_type) body = OpenRLHFRewardResponse(rewards=rewards, error_types=error_types) # FastAPI's own rendering writes a NaN as null, or refuses it, by version. return Response(body.model_dump_json(), media_type='application/json') diff --git a/deepgym/src/deepgym/models.py b/deepgym/src/deepgym/models.py index 39d803a..cde43de 100644 --- a/deepgym/src/deepgym/models.py +++ b/deepgym/src/deepgym/models.py @@ -67,13 +67,18 @@ def checked_score(value: object) -> float: def finite_components(components: dict[str, float] | None) -> dict[str, float] | None: - """Replace non-finite reward components with 0.0 so they never reach a trainer.""" - if components is None: - return None - bad = [key for key, value in components.items() if not math.isfinite(value)] + """Return *components* if every one is finite. + + A NaN or infinite component is a verifier bug, like an invalid score: zeroing it + would hand a trainer a fabricated 0 under an otherwise valid verdict. + + Raises: + InvalidScoreError: If a component is NaN or infinite. + """ + bad = [key for key, value in (components or {}).items() if not math.isfinite(value)] if bad: - logger.warning('Non-finite reward components %s set to 0.0', bad) - return {key: (0.0 if key in bad else value) for key, value in components.items()} + raise InvalidScoreError(f'Verifier returned non-finite reward components {bad}') + return components VerifierMode = Literal['auto', 'script', 'verify_function', 'body'] @@ -252,13 +257,15 @@ def zero_non_finite_time(cls, value: float) -> float: @model_validator(mode='before') @classmethod - def zero_invalid_score(cls, data: object) -> object: - """Fail a case whose score is NaN, infinite, or out of range instead of crashing.""" + def check_score(cls, data: object) -> object: + """Clamp float noise; reject a NaN, infinite or out-of-range score. + + Raises: + InvalidScoreError: Never a 0: that would be a trainable reward the verifier + didn't give, under an otherwise valid verdict. + """ if isinstance(data, dict) and 'score' in data: - try: - return {**data, 'score': checked_score(data['score'])} - except InvalidScoreError: - return {**data, 'score': 0.0, 'passed': False, 'error': 'invalid score'} + return {**data, 'score': checked_score(data['score'])} return data @model_validator(mode='after') @@ -292,8 +299,8 @@ class VerifierResult(BaseModel): @field_validator('reward_components') @classmethod - def zero_non_finite_components(cls, value: dict[str, float] | None) -> dict[str, float] | None: - """Replace NaN/inf components with 0.0 before they reach a trainer.""" + def check_components(cls, value: dict[str, float] | None) -> dict[str, float] | None: + """Reject NaN/inf components before they reach a trainer.""" return finite_components(value) metrics: dict[str, Any] | None = None @@ -341,8 +348,8 @@ class RunResult(BaseModel): @field_validator('reward_components') @classmethod - def zero_non_finite_components(cls, value: dict[str, float] | None) -> dict[str, float] | None: - """Replace NaN/inf components with 0.0 before they reach a trainer.""" + def check_components(cls, value: dict[str, float] | None) -> dict[str, float] | None: + """Reject NaN/inf components before they reach a trainer.""" return finite_components(value) metrics: dict[str, Any] | None = None diff --git a/deepgym/src/deepgym/reward_qa.py b/deepgym/src/deepgym/reward_qa.py index 794bc20..b04cae5 100644 --- a/deepgym/src/deepgym/reward_qa.py +++ b/deepgym/src/deepgym/reward_qa.py @@ -2,7 +2,9 @@ from __future__ import annotations +import contextlib import hashlib +from collections.abc import Iterator from pathlib import Path from typing import Literal @@ -195,7 +197,7 @@ def __init__( db: ExploitDB | None = None, ) -> None: self._tester = AdversarialTester(dg, pass_threshold=pass_threshold) - self._db = db + self._db_given = db def audit( self, @@ -208,18 +210,28 @@ def audit( db_path: Path | None = None, ) -> VerifierAuditReport: """Run a verifier audit and return a structured QA report.""" - report = self._tester.test(env, strategies=strategies) verifier_source = env.verifier_source resolved_verifier_id = verifier_id or env.task[:80] verifier_hash = fingerprint_verifier(verifier_source) + if persist: + # Before the retest, as the scanners do: a stored result for a changed verifier + # is dropped, or an inconclusive retest (never saved) would leave it standing. + with self._db(db_path) as db: + db.current( + resolved_verifier_id, + benchmark=benchmark, + attack_type='white-box', + verifier_hash=verifier_hash, + attack_config='', + ) + report = self._tester.test(env, strategies=strategies) patterns = classify_patterns(verifier_source, report) status = audit_status(report) risk_score = None if status == 'inconclusive' else compute_risk_score(patterns, report) stored = False if persist: - db, should_close = self._open_db(db_path) - try: + with self._db(db_path) as db: # False for an inconclusive audit: it never judged the verifier. stored = db.save( build_exploit_record( @@ -230,9 +242,6 @@ def audit( report=report, ) ) - finally: - if should_close: - db.close() return VerifierAuditReport( verifier_id=resolved_verifier_id, @@ -251,7 +260,14 @@ def audit( stored=stored, ) - def _open_db(self, db_path: Path | None) -> tuple[ExploitDB, bool]: - if self._db is not None: - return self._db, False - return ExploitDB(db_path=db_path), True + @contextlib.contextmanager + def _db(self, db_path: Path | None) -> Iterator[ExploitDB]: + """The auditor's database if it was given one, else one opened (and closed) here.""" + if self._db_given is not None: + yield self._db_given + return + db = ExploitDB(db_path=db_path) + try: + yield db + finally: + db.close() diff --git a/deepgym/src/deepgym/traces/export.py b/deepgym/src/deepgym/traces/export.py index 2972e46..eee01e7 100644 --- a/deepgym/src/deepgym/traces/export.py +++ b/deepgym/src/deepgym/traces/export.py @@ -8,12 +8,14 @@ * ``rl``: one conversation per trace, for SFT and RL trainers. ``messages`` concatenates the model messages recorded at each step (TRL's conversational format, the ``messages`` key of verl's multi-turn SFT dataset), with token ids and logprobs where - recorded; ``reward`` and ``passed`` come from the final verifier; ``step_rewards`` - are the step verifier's scores (null where none ran). ``steps`` holds every action in - order with what it produced (text, base64 PNG screenshot, done flag, exit code, or the - error if nothing), its step reward, and ``message_index``: where in ``messages`` the - reply that produced it is, shared by the actions of one turn. A trace with a turn that - has no model messages has no whole conversation to train on and is skipped. + recorded; ``initial_observation`` and ``initial_screenshot`` (base64 PNG) are what + the episode showed before the first action; ``reward`` and ``passed`` come from the + final verifier; ``step_rewards`` are the step verifier's scores (null where none + ran). ``steps`` holds every action in order with what it produced (text, base64 PNG + screenshot, done flag, exit code, or the error if nothing), its step reward, and + ``message_index``: where in ``messages`` the reply that produced it is, shared by the + actions of one turn. A trace with a turn that has no model messages has no whole + conversation to train on and is skipped. Every row carries ``has_verdict``, and its ``score``/``reward`` and ``passed`` are null without one, so a failed sandbox or verifier never reads as a scored 0. ``only_valid`` @@ -61,6 +63,7 @@ def rl_row(trace: Trace) -> Row | None: return None verdict = trace.result if trace.has_verdict else None provenance = trace.provenance + start = trace.initial_observation messages: list[JsonValue] = [] steps: list[JsonValue] = [] for step in trace.steps: @@ -84,6 +87,8 @@ def rl_row(trace: Trace) -> Row | None: 'agent': provenance.agent.name, 'model': provenance.agent.model, 'messages': messages, + 'initial_observation': None if start is None else start.content, + 'initial_screenshot': None if start is None else start.screenshot, 'reward': None if verdict is None else verdict.score, 'passed': None if verdict is None else verdict.passed, 'step_rewards': [step.reward for step in trace.steps], @@ -236,6 +241,8 @@ def _schema(rows: RowShape) -> pa.Schema: ('agent', pa.string()), ('model', pa.string()), ('messages', pa.list_(message)), + ('initial_observation', pa.string()), + ('initial_screenshot', pa.string()), ('reward', pa.float64()), ('passed', pa.bool_()), ('step_rewards', pa.list_(pa.float64())), diff --git a/deepgym/src/deepgym/verifier_template.py b/deepgym/src/deepgym/verifier_template.py index 47d28a3..001968c 100644 --- a/deepgym/src/deepgym/verifier_template.py +++ b/deepgym/src/deepgym/verifier_template.py @@ -29,13 +29,28 @@ _REPORT = textwrap.dedent('''\ import json as _dg_json + import math as _dg_math import os as _dg_os import sys as _dg_sys import traceback as _dg_traceback class _DgInvalidScore(ValueError): - """The verifier returned NaN, inf, or a score outside [0, 1].""" + """The verifier returned NaN, inf, or a score or case score outside [0, 1].""" + + + def _dg_sub_score(value, what, bounded): + """*value* as a float: a finite number, within [0, 1] (up to float noise) if *bounded*. + + The float is what gets printed, so the host never reads the value differently. + """ + try: + number = float(value) + except (TypeError, ValueError): + raise _DgInvalidScore(f"{what} {value!r} is not a number") from None + if not _dg_math.isfinite(number) or bounded and not -1e-6 <= number <= 1 + 1e-6: + raise _DgInvalidScore(f"invalid {what} {value!r}") + return number def _dg_report(run): @@ -83,6 +98,18 @@ def _dg_report(run): # Clamp only float noise. NaN fails this check too; never turn it into reward. if not -1e-6 <= output["score"] <= 1 + 1e-6: raise _DgInvalidScore(f"invalid score {output['score']!r}; must be in [0, 1]") + # Sub-scores too, as the host checks them, so one the solution's output made + # invalid is its judged 0 here rather than a run with no verdict there. + for case in output.get("cases") or (): + if not isinstance(case, dict): + raise _DgInvalidScore(f"case {case!r} is not an object") + if "score" in case: + case["score"] = _dg_sub_score(case["score"], "case score", True) + if output.get("reward_components") is not None: + output["reward_components"] = { + name: _dg_sub_score(value, f"reward component {name!r}", False) + for name, value in output["reward_components"].items() + } output["score"] = max(0.0, min(1.0, output["score"])) output.setdefault("truncated", False) # A line of its own, even after a partial line the verifier left unterminated. diff --git a/deepgym/tests/golden/traces/rl.jsonl b/deepgym/tests/golden/traces/rl.jsonl index b2e57b6..dfb0371 100644 --- a/deepgym/tests/golden/traces/rl.jsonl +++ b/deepgym/tests/golden/traces/rl.jsonl @@ -1,2 +1,2 @@ -{"trace_id":"trace0001","data_source":"write_notes","agent":"react-shell","model":"Qwen/Qwen3-8B","messages":[{"role":"system","content":"You are a shell agent."},{"role":"user","content":"Write hello to notes/out.txt, then finish.\n$ "},{"role":"assistant","content":"mkdir -p notes && echo hello > notes/out.txt","token_ids":[11,12,13],"logprobs":[-0.25,-0.5,-0.125]},{"role":"tool","content":"exit 0"},{"role":"assistant","content":"write notes/extra.txt"},{"role":"user","content":"write_file is not allowed."},{"role":"assistant","content":"done"}],"reward":1.0,"passed":true,"step_rewards":[0.5,null,0.5],"steps":[{"message_index":2,"action":{"action_type":"bash","content":"mkdir -p notes && echo hello > notes/out.txt","x":null,"y":null,"to_x":null,"to_y":null,"path":null,"amount":1},"observation":"","screenshot":null,"done":false,"exit_code":0,"error":null,"reward":0.5},{"message_index":4,"action":{"action_type":"write_file","content":"x","x":null,"y":null,"to_x":null,"to_y":null,"path":"notes/extra.txt","amount":1},"observation":null,"screenshot":null,"done":null,"exit_code":null,"error":"ActionError: Action 'write_file' not allowed here.","reward":null},{"message_index":6,"action":{"action_type":"done","content":"","x":null,"y":null,"to_x":null,"to_y":null,"path":null,"amount":1},"observation":"","screenshot":null,"done":true,"exit_code":null,"error":null,"reward":0.5}],"has_verdict":true} -{"trace_id":"trace0003","data_source":"write_notes","agent":"react-shell","model":"Qwen/Qwen3-8B","messages":[{"role":"system","content":"You are a shell agent."},{"role":"user","content":"Write hello to notes/out.txt, then finish.\n$ "},{"role":"assistant","content":"mkdir -p notes && echo hello > notes/out.txt","token_ids":[11,12,13],"logprobs":[-0.25,-0.5,-0.125]},{"role":"tool","content":"exit 0"},{"role":"assistant","content":"write notes/extra.txt"},{"role":"user","content":"write_file is not allowed."},{"role":"assistant","content":"done"}],"reward":null,"passed":null,"step_rewards":[0.5,null,0.5],"steps":[{"message_index":2,"action":{"action_type":"bash","content":"mkdir -p notes && echo hello > notes/out.txt","x":null,"y":null,"to_x":null,"to_y":null,"path":null,"amount":1},"observation":"","screenshot":null,"done":false,"exit_code":0,"error":null,"reward":0.5},{"message_index":4,"action":{"action_type":"write_file","content":"x","x":null,"y":null,"to_x":null,"to_y":null,"path":"notes/extra.txt","amount":1},"observation":null,"screenshot":null,"done":null,"exit_code":null,"error":"ActionError: Action 'write_file' not allowed here.","reward":null},{"message_index":6,"action":{"action_type":"done","content":"","x":null,"y":null,"to_x":null,"to_y":null,"path":null,"amount":1},"observation":"","screenshot":null,"done":true,"exit_code":null,"error":null,"reward":0.5}],"has_verdict":false} +{"trace_id":"trace0001","data_source":"write_notes","agent":"react-shell","model":"Qwen/Qwen3-8B","messages":[{"role":"system","content":"You are a shell agent."},{"role":"user","content":"Write hello to notes/out.txt, then finish.\n$ "},{"role":"assistant","content":"mkdir -p notes && echo hello > notes/out.txt","token_ids":[11,12,13],"logprobs":[-0.25,-0.5,-0.125]},{"role":"tool","content":"exit 0"},{"role":"assistant","content":"write notes/extra.txt"},{"role":"user","content":"write_file is not allowed."},{"role":"assistant","content":"done"}],"initial_observation":"$ ","initial_screenshot":"iVBORw0KGgogc2FtcGxlIHNjcmVlbnNob3Q=","reward":1.0,"passed":true,"step_rewards":[0.5,null,0.5],"steps":[{"message_index":2,"action":{"action_type":"bash","content":"mkdir -p notes && echo hello > notes/out.txt","x":null,"y":null,"to_x":null,"to_y":null,"path":null,"amount":1},"observation":"","screenshot":null,"done":false,"exit_code":0,"error":null,"reward":0.5},{"message_index":4,"action":{"action_type":"write_file","content":"x","x":null,"y":null,"to_x":null,"to_y":null,"path":"notes/extra.txt","amount":1},"observation":null,"screenshot":null,"done":null,"exit_code":null,"error":"ActionError: Action 'write_file' not allowed here.","reward":null},{"message_index":6,"action":{"action_type":"done","content":"","x":null,"y":null,"to_x":null,"to_y":null,"path":null,"amount":1},"observation":"","screenshot":null,"done":true,"exit_code":null,"error":null,"reward":0.5}],"has_verdict":true} +{"trace_id":"trace0003","data_source":"write_notes","agent":"react-shell","model":"Qwen/Qwen3-8B","messages":[{"role":"system","content":"You are a shell agent."},{"role":"user","content":"Write hello to notes/out.txt, then finish.\n$ "},{"role":"assistant","content":"mkdir -p notes && echo hello > notes/out.txt","token_ids":[11,12,13],"logprobs":[-0.25,-0.5,-0.125]},{"role":"tool","content":"exit 0"},{"role":"assistant","content":"write notes/extra.txt"},{"role":"user","content":"write_file is not allowed."},{"role":"assistant","content":"done"}],"initial_observation":"$ ","initial_screenshot":"iVBORw0KGgogc2FtcGxlIHNjcmVlbnNob3Q=","reward":null,"passed":null,"step_rewards":[0.5,null,0.5],"steps":[{"message_index":2,"action":{"action_type":"bash","content":"mkdir -p notes && echo hello > notes/out.txt","x":null,"y":null,"to_x":null,"to_y":null,"path":null,"amount":1},"observation":"","screenshot":null,"done":false,"exit_code":0,"error":null,"reward":0.5},{"message_index":4,"action":{"action_type":"write_file","content":"x","x":null,"y":null,"to_x":null,"to_y":null,"path":"notes/extra.txt","amount":1},"observation":null,"screenshot":null,"done":null,"exit_code":null,"error":"ActionError: Action 'write_file' not allowed here.","reward":null},{"message_index":6,"action":{"action_type":"done","content":"","x":null,"y":null,"to_x":null,"to_y":null,"path":null,"amount":1},"observation":"","screenshot":null,"done":true,"exit_code":null,"error":null,"reward":0.5}],"has_verdict":false} diff --git a/deepgym/tests/test_adversarial.py b/deepgym/tests/test_adversarial.py index 15bf1fd..7c6d327 100644 --- a/deepgym/tests/test_adversarial.py +++ b/deepgym/tests/test_adversarial.py @@ -401,3 +401,12 @@ def test_no_attacks_is_never_robust() -> None: env = Environment(task='Anything', verifier_code='return 0.0\n') assert not AdversarialReport.from_results(env, []).is_robust + + +def test_an_unknown_strategy_is_rejected_before_any_attack_runs(dg: DeepGym) -> None: + """A misspelt strategy used to run a no-op and report the verifier robust.""" + tester = AdversarialTester(dg) + tester._execute_attack = lambda *args: pytest.fail('an attack ran') + + with pytest.raises(ValueError, match='overlfow'): + tester.test(Environment(task='t', verifier_code=ROBUST_VERIFIER), ['empty', 'overlfow']) diff --git a/deepgym/tests/test_api.py b/deepgym/tests/test_api.py index 96d88da..fdb35c6 100644 --- a/deepgym/tests/test_api.py +++ b/deepgym/tests/test_api.py @@ -262,6 +262,13 @@ def test_audit_registered_environment(self, client: TestClient) -> None: assert data['verifier_id'] == env_id assert data['exploitable'] is True + def test_audit_rejects_an_unknown_strategy(self, client: TestClient) -> None: + payload = { + 'environment': {'task': 'Do anything', 'verifier_code': STANDALONE_PASSING_VERIFIER}, + 'strategies': ['overlfow'], + } + assert client.post('/v1/audit/verifier', json=payload).status_code == 422 + class TestBenchmarkAuditEndpoint: """Test benchmark split leakage auditing for registered environments.""" diff --git a/deepgym/tests/test_harbor_export.py b/deepgym/tests/test_harbor_export.py index b0f110f..82d896e 100644 --- a/deepgym/tests/test_harbor_export.py +++ b/deepgym/tests/test_harbor_export.py @@ -165,6 +165,16 @@ def test_export_refuses_envs_it_cannot_express(tmp_path: Path, env: Environment) export_environment(env, tmp_path) +@pytest.mark.parametrize('image', [None, 'python:3.12-slim']) +def test_export_refuses_a_snapshot_runtime(tmp_path: Path, image: str | None) -> None: + """Its dependencies would silently vanish from a plain image.""" + env = load_environment('two_sum').model_copy(update={'snapshot': 'numpy-env', 'image': image}) + + with pytest.raises(DeepGymError, match='snapshot'): + export_environment(env, tmp_path) + assert not (tmp_path / 'two_sum').exists() + + def test_export_never_overwrites(tmp_path: Path) -> None: env = load_environment('two_sum') export_environment(env, tmp_path) diff --git a/deepgym/tests/test_integrations.py b/deepgym/tests/test_integrations.py index e5bd8fc..58cc891 100644 --- a/deepgym/tests/test_integrations.py +++ b/deepgym/tests/test_integrations.py @@ -16,7 +16,7 @@ from deepgym.async_core import AsyncDeepGym from deepgym.core import DeepGym -from deepgym.exceptions import NoVerdictError, SandboxError, VerifierError +from deepgym.exceptions import InvalidScoreError, NoVerdictError, SandboxError, VerifierError from deepgym.exceptions import TimeoutError as DeepGymTimeout from deepgym.integrations import openrlhf from deepgym.integrations import verl as verl_module @@ -469,7 +469,8 @@ def _post( class _ScriptedDeepGym(DeepGym): """A client that runs nothing: each run of an output does the next step of its script. - A step that is an exception is raised; a number is seconds to wait before judging. + A step that is an exception is raised, a RunResult returned; a number is seconds to + wait before judging. Every run first waits *seconds*. A judged output scores 1.0 if it says 'right', else 0.0. The client records each run's output, the most runs going at once, and when the first run started. @@ -498,6 +499,8 @@ def run(self, env: Environment, model_output: str, **kwargs: object) -> RunResul time.sleep(self.seconds) if isinstance(step, BaseException): raise step + if isinstance(step, RunResult): + return step time.sleep(step) judged = 'right' in model_output return RunResult( @@ -601,7 +604,8 @@ def test_score_endpoint_never_scores_prompt_text( def test_failures_the_model_triggers_never_fail_the_request(self, local_dg: DeepGym) -> None: """A failed request stops OpenRLHF's training step, so no output may cause one. - The verifier crashes on 'boom', and the last query lost its prompt: each scores 0. + The verifier reports its own error on 'boom', and the last query lost its prompt: + neither has a verdict, so each gets the infrastructure score (0 here). """ payload = { 'prompts': [OPENRLHF_PROMPT] * 3, @@ -615,13 +619,59 @@ def test_failures_the_model_triggers_never_fail_the_request(self, local_dg: Deep 'error_types': [None, 'runtime_error', 'prompt_mismatch'], } - def test_only_provider_failures_get_nan(self, local_dg: DeepGym) -> None: - """NaN drops the sample's group; a verifier the model's output broke must score 0.""" + def test_a_verifier_error_gets_nan_not_a_trainable_zero(self, local_dg: DeepGym) -> None: + """No verdict is never a failed answer: NaN drops the sample's group.""" payload = {'prompts': ['p'], 'outputs': ['boom']} data = _post(MIXED, local_dg, payload, infra_failure_score='nan') - assert data == {'rewards': [0.0], 'error_types': ['runtime_error']} + assert math.isnan(data['rewards'][0]) + assert data['error_types'] == ['runtime_error'] + + def test_no_failure_is_rewarded_above_zero(self, monkeypatch: pytest.MonkeyPatch) -> None: + """Every output says 'right', yet each run fails: none may earn more than 0. + + Failures without a verdict (provider, invalid score, verifier) get NaN; the model's + own judged failure and a run still going at the deadline score 0. + """ + monkeypatch.setattr(openrlhf, '_RETRY_DELAY_S', 0.0) + monkeypatch.setattr(openrlhf, '_REQUEST_BUDGET_S', 0.5) + fault = RunResult( + score=0.0, + passed=False, + output='', + stderr='', + exit_code=1, + execution_time_ms=0.0, + sandbox_id='fake', + error_type='solution_error', + ) + script: dict[str, list[BaseException | float | RunResult]] = { + 'right outage': [ConnectionError('down'), ConnectionError('down')], + 'right invalid': [InvalidScoreError('score nan')], + 'right verifier': [VerifierError('no JSON')], + 'right fault': [fault], + 'right slow': [2.0], + } + outputs = list(script) + + data = _post( + MIXED, _ScriptedDeepGym(script), {'prompts': outputs, 'outputs': outputs}, 'nan' + ) + + rewards = dict(zip(outputs, data['rewards'])) + assert not any(reward > 0 for reward in rewards.values()) + assert [math.isnan(rewards[o]) for o in outputs] == [True, True, True, False, False] + assert (rewards['right fault'], rewards['right slow']) == (0.0, 0.0) + + def test_a_judged_model_fault_scores_zero_even_with_nan(self, local_dg: DeepGym) -> None: + """The model's own failure has a verdict, so it can't dodge its 0 as a NaN.""" + verdict = {'score': 0.0, 'passed': False, 'error_type': 'solution_error'} + env = Environment(task='t', verifier_code=f'return {verdict!r}\n') + + data = _post(env, local_dg, {'prompts': ['p'], 'outputs': ['x']}, infra_failure_score='nan') + + assert data == {'rewards': [0.0], 'error_types': ['solution_error']} @pytest.mark.parametrize( 'failure', @@ -654,10 +704,10 @@ def test_a_provider_failure_is_graded_once_more( ], ids=['sandbox_killed', 'hung', 'verifier_failed', 'sdk_cancelled'], ) - def test_a_failure_the_model_may_have_caused_is_never_graded_again( + def test_a_failure_other_than_the_providers_is_never_graded_again( self, monkeypatch: pytest.MonkeyPatch, failure: BaseException ) -> None: - """It scores 0, even with NaN asked for: a second grading could hand it a pass. + """It has no verdict, so NaN, never a trainable 0; a second grading could hand it a pass. So does asyncio's CancelledError, which a provider SDK running its own event loop can raise and which isn't an Exception: it never fails the request. @@ -667,7 +717,7 @@ def test_a_failure_the_model_may_have_caused_is_never_graded_again( data = _post(MIXED, dg, {'prompts': ['p'], 'outputs': ['right']}, infra_failure_score='nan') - assert data['rewards'] == [0.0] + assert math.isnan(data['rewards'][0]) assert dg.calls == ['right'] @pytest.mark.parametrize('infra_failure_score', ['zero', 'nan']) diff --git a/deepgym/tests/test_per_test_traces.py b/deepgym/tests/test_per_test_traces.py index 503c0eb..f7af785 100644 --- a/deepgym/tests/test_per_test_traces.py +++ b/deepgym/tests/test_per_test_traces.py @@ -10,6 +10,7 @@ import pytest from deepgym.core import DeepGym +from deepgym.exceptions import InvalidScoreError from deepgym.gym import DeepGymEnv, GymInfo from deepgym.integrations.reward import RewardFunction from deepgym.models import ( @@ -61,10 +62,10 @@ def test_full_creation(self) -> None: assert tc.input_summary == 'coins=[1,5], amount=11' @pytest.mark.parametrize('score', [-0.1, 1.1, float('nan')]) - def test_invalid_score_fails_the_case(self, score: float) -> None: - """An invalid case score marks the case failed with score 0 instead of crashing.""" - case = CaseResult(passed=True, score=score) - assert (case.score, case.passed) == (0.0, False) + def test_invalid_score_is_rejected(self, score: float) -> None: + """An invalid case score is a verifier bug, never a trainable 0.""" + with pytest.raises(InvalidScoreError): + CaseResult(passed=True, score=score) def test_failed_case_without_score_gets_no_credit(self) -> None: """A failed case scores 0 unless the verifier gave it an explicit partial score.""" diff --git a/deepgym/tests/test_reward_qa.py b/deepgym/tests/test_reward_qa.py index 5180c34..4caff64 100644 --- a/deepgym/tests/test_reward_qa.py +++ b/deepgym/tests/test_reward_qa.py @@ -9,7 +9,7 @@ from deepgym.exploit_db import ExploitDB from deepgym.models import Environment from deepgym.reward_qa import RewardAuditor, fingerprint_verifier -from tests.test_adversarial import CRASHING_VERIFIER +from tests.test_adversarial import CRASHING_VERIFIER, ROBUST_VERIFIER STANDALONE_PASSING_VERIFIER = ( 'import sys, json\n' @@ -73,6 +73,30 @@ def test_inconclusive_audits_are_never_stored_as_tested( finally: db.close() + def test_an_inconclusive_retest_of_a_changed_verifier_drops_its_old_result( + self, tmp_path: Path, dg: DeepGym + ) -> None: + """The old verifier's robust result must not stand in for the changed one.""" + db_path = tmp_path / 'audits.db' + auditor = RewardAuditor(dg) + reports = [ + auditor.audit( + Environment(task='Write compute(a, b).', verifier_code=verifier), + verifier_id='v', + strategies=['empty'], + persist=True, + db_path=db_path, + ) + for verifier in (ROBUST_VERIFIER, CRASHING_VERIFIER) + ] + + assert [report.status for report in reports] == ['robust', 'inconclusive'] + db = ExploitDB(db_path=db_path) + try: + assert db.get('v', benchmark='custom') is None + finally: + db.close() + def test_audit_persists_to_exploit_db(self, tmp_path: Path, dg: DeepGym) -> None: env = Environment(task='Do anything', verifier_code=STANDALONE_PASSING_VERIFIER) auditor = RewardAuditor(dg) diff --git a/deepgym/tests/test_score_validation.py b/deepgym/tests/test_score_validation.py index e2c4367..47607c6 100644 --- a/deepgym/tests/test_score_validation.py +++ b/deepgym/tests/test_score_validation.py @@ -71,17 +71,27 @@ def test_float_noise_is_clamped(dg: DeepGym, make_env) -> None: assert result.score == 1.0 -def test_non_finite_reward_components_become_zero(dg: DeepGym) -> None: - env = _script('0.5', '{"good": 0.5, "nan": float("nan"), "inf": float("inf")}') - result = dg.run(env, '') - assert result.reward_components == {'good': 0.5, 'nan': 0.0, 'inf': 0.0} - shaped = make_dapo_shaped_reward_fn(env, dg, component='nan') - assert shaped(completions=['']) == [0.0] +@pytest.mark.parametrize('bad', ['float("nan")', 'float("inf")']) +def test_non_finite_reward_components_void_the_verdict(dg: DeepGym, bad: str) -> None: + """A valid overall score never vouches for a broken component: no trainable 0.""" + env = _script('1.0', f'{{"good": 0.5, "bad": {bad}}}') + with pytest.raises(InvalidScoreError): + dg.run(env, '') + (result,) = dg.run_batch(env, ['']).results + assert (result.has_verdict, result.error_type) == (False, 'invalid_score') + assert make_dapo_shaped_reward_fn(env, dg, component='bad')(completions=['']) == [None] -def test_invalid_case_score_fails_that_case() -> None: - case = CaseResult(id='c', passed=True, score=float('nan')) - assert (case.score, case.passed) == (0.0, False) +@pytest.mark.parametrize('score', ['float("nan")', '-1', '7.5']) +def test_invalid_case_scores_void_the_verdict(dg: DeepGym, score: str) -> None: + case = f'{{"passed": True, "score": {score}}}' + env = Environment( + task='t', verifier_code=f'return {{"score": 1.0, "passed": True, "cases": [{case}]}}\n' + ) + with pytest.raises(InvalidScoreError): + dg.run(env, '') + with pytest.raises(InvalidScoreError): + CaseResult(id='c', passed=True, score=float('nan')) @pytest.mark.parametrize('value', [math.nan, math.inf, -math.inf, -1, 7.5, 'abc']) diff --git a/deepgym/tests/test_traces_export.py b/deepgym/tests/test_traces_export.py index 8343470..96da6aa 100644 --- a/deepgym/tests/test_traces_export.py +++ b/deepgym/tests/test_traces_export.py @@ -184,6 +184,12 @@ def test_rl_rows_keep_screenshots(self, tmp_path: Path, fmt: str) -> None: (row,) = _rows(tmp_path / f'rl.{fmt}', fmt) assert [step['screenshot'] for step in row['steps']] == [shot, None, None] assert [step['done'] for step in row['steps']] == [False, None, True] + # The screenshot the first action answered to. + start = trace.initial_observation + assert (row['initial_observation'], row['initial_screenshot']) == ( + start.content, + start.screenshot, + ) def test_only_valid_drops_unjudged_traces(self, tmp_path: Path) -> None: out = tmp_path / 'valid.jsonl' diff --git a/deepgym/tests/test_verifier_template.py b/deepgym/tests/test_verifier_template.py index 7bc008d..9a6a083 100644 --- a/deepgym/tests/test_verifier_template.py +++ b/deepgym/tests/test_verifier_template.py @@ -179,6 +179,26 @@ def test_float_noise_is_clamped(self) -> None: output, _ = _run_wrapped('return 1.0000000001\n', '# e\n') assert output['score'] == 1.0 + @pytest.mark.parametrize( + 'extra', ['"cases": [{"passed": True, "score": None}]', '"reward_components": {"c": "x"}'] + ) + def test_a_non_numeric_sub_score_is_an_invalid_score(self, extra: str) -> None: + output, code = _run_wrapped(f'return {{"score": 1.0, "passed": True, {extra}}}\n', '') + assert (output['score'], output['error_type'], code) == (0.0, 'invalid_score', 2) + + def test_sub_scores_are_printed_as_the_numbers_checked(self) -> None: + """float() reads an Arabic-Indic one as 1.0; the host must get 1.0, not the text.""" + body = ( + 'return {"score": 1.0, "passed": True, "reward_components": {"c": "١"}, ' + '"cases": [{"passed": True, "score": "0.5"}]}\n' + ) + output, code = _run_wrapped(body, '') + assert (output['reward_components'], output['cases'][0]['score'], code) == ( + {'c': 1.0}, + 0.5, + 0, + ) + def _crash_after(score: float, mode: str) -> str: """A verifier that prints *score* as a failed run, then crashes outside any handler.""" @@ -244,6 +264,41 @@ def test_an_invalid_result_is_the_solutions_once_it_ran( run = DeepGym(mode='local').run(env, 'x = 1\n') assert (run.has_verdict, run.score, run.error_type) == (True, 0.0, 'solution_error') + @pytest.mark.parametrize('mode', ['body', 'verify_function']) + @pytest.mark.parametrize( + 'extra', + [ + '"cases": [{"passed": True, "score": value}]', + '"cases": [value]', + '"reward_components": {"style": value}', + ], + ids=['case_score', 'bare_case', 'component'], + ) + @pytest.mark.parametrize('consulted', [False, True], ids=['own_constant', 'after_the_solution']) + def test_an_invalid_sub_score_is_the_solutions_once_it_ran( + self, mode: str, extra: str, consulted: bool + ) -> None: + """A NaN sub-score under a valid overall score can't mask the solution's 0.""" + value = 'load_solution(solution_path).answer()' if consulted else 'float("nan")' + body = ( + ' from deepgym_solution import load_solution\n' + f' value = {value}\n' + f' return {{"score": 1.0, "passed": True, {extra}}}\n' + ) + if mode == 'verify_function': + body = f'def verify(solution_path, test_cases_path=None):\n{body}' + else: + body = body.replace('\n ', '\n').removeprefix(' ') + env = Environment(task='t', verifier_code=body, verifier_mode=mode) + solution = 'def answer():\n return float("nan")\n' + + if not consulted: # the verifier's own constant: no verdict + with pytest.raises(InvalidScoreError): + DeepGym(mode='local').run(env, solution) + return + run = DeepGym(mode='local').run(env, solution) + assert (run.has_verdict, run.score, run.error_type) == (True, 0.0, 'solution_error') + @pytest.mark.parametrize( ('shape', 'mode'), [