diff --git a/deepgym/CHANGELOG.md b/deepgym/CHANGELOG.md index 4af16db..01d3699 100644 --- a/deepgym/CHANGELOG.md +++ b/deepgym/CHANGELOG.md @@ -2,6 +2,7 @@ ## Unreleased +- **Sessions: no verdict only for failures that are ours for certain.** Once the agent has acted, a failure is its judged 0 (`policy_sandbox_failure`) unless it is ours for certain: a network, provider or auth failure (HTTP 401 and 403 now count as infrastructure in `is_infrastructure_failure`), a verifier that couldn't run (its upload or launch failed) or timed out before it read the agent's output, an interrupt, or E2B or Modal retiring the sandbox at the end of its one-hour lifetime (new `SandboxExpiredError`, before a `SandboxDiedError`) unless the agent's own actions took at least half the session's time. A failure the rule can't read is the agent's 0. The rule is `deepgym.session.failure_result`, which verl's `deepgym_agent` loop and Harbor's agent phase also use; before, a 401 uploading verifier code or a verl final verifier that timed out before reading anything scored 0, and a Harbor agent that replaced its work directory with a link, grew it past the archive cap, made packing outrun its deadline, killed its sandbox or spent most of the provider lifetime in its script got no verdict; in Harbor any failure once the agent's script has started is now its 0 unless it is ours for certain. Session verifiers now get the consulted mark single runs use, embedded in the verifier's own script rather than on a command line: a step or final verifier that times out after reading the agent's output (a FIFO it left, say) is a judged 0 with `solution_timeout` (for a step, its reward, and the episode goes on), and so is a verifier that is killed, or gives no verdict, after reading it (`solution_error`), unless the agent's code failed to launch. Each verifier also prints a start line first, so a timeout whose output the provider lost is a judged 0 rather than passing for one that never read anything. Listing the agent's folder, or starting a program in it or on one of its files, now counts as reading its output, and the mark is written to the stdout file itself, so capturing `sys.stdout` can't hide it, and cut from what the verifier printed exactly, so a line it fell inside reads whole. Daytona enforces command deadlines in the sandbox, so a timed-out command keeps what it printed. A FIFO the agent leaves where the harness writes is refused at once instead of hanging the write, the local backend shows output that isn't UTF-8 instead of ending the episode, and E2B, Daytona and Docker/Apple uploads to a stopped sandbox raise `SandboxDiedError`. `deepgym_solution.run_solution` prints the consulted mark before it releases the solution, so a solution that dies the moment it starts can't pass for one that never ran. - **QA gate:** multi-turn and computer-use envs with a `reference_actions.json` are checked: the reference replays at full marks every time, and agents that quit at once (`empty_fails`), use every action (`busy_fails`) or act at random (`random_fails`) must earn nothing. The computer-use built-ins ship their reference episodes (one per variant) and run on `--mode daytona`. Negative controls (empty, stub, idle agents) fail on any reward, not just full marks; attacks define the env's real entry point; reports list the checks that `ran`. `difficulty` is reported, never gated: it gives mean and spread of rewards and flags `too_easy`, `too_hard` and `no_variance` (partial rewards count as variance), retries 429s, timeouts and 5xx with backoff, and is inconclusive rather than aborting when the endpoint stays down. Missing verdicts never pass: incomplete black-box scans are `inconclusive` and never saved as ROBUST, an audit that judged nothing has no risk rating (`status` says why), and RL exploit discovery reports suspected exploits with their signs from every round instead of a genuineness verdict. The exploit DB keys results by benchmark, verifier and attack mode (legacy tables are rebuilt in place), reuses one only for the same verifier contents and attack setup, and summaries filter by benchmark and mode. Reports and approvals are written atomically, and attack payloads carry JSON test values as valid Python. Built-ins: `data_pipeline` and `web_path_traversal_toy` no longer pay for an empty answer, and `anagram_check`, `palindrome_check` and `valid_parentheses` score balanced recall on real booleans, so a constant answer earns nothing. Breaking: `ExploitDB.get` needs `benchmark=`, `DiscoveryReport` replaces `is_genuine_solution`/`exploit_discovered` with `suspected_exploits`/`exploit_suspected`, and `VerifierAuditReport.risk_score`/`risk_level` are None when inconclusive. - **Environment authoring:** `deepgym new --kind function|script|repo-patch|multi-turn` scaffolds a working env folder (task, a verifier that runs the solution only through `deepgym_solution` and reports every case with a reason, reference, seeded `variants.py`, metadata). `deepgym check ` validates the files, metadata schema and protocol output, runs the reference, empty and stub solutions, and prints warnings from an advisory lint over the judge code (never evidence of isolation, which comes from `load_solution`/`run_solution`). `deepgym qa` takes folders (multi-turn ones replay `reference_actions.json`); `--mutants` fails a verifier that passes any AST mutant of the reference and reports the mutation score, running each mutant's whole judge confined in a local bwrap sandbox (`DeepGym(confine_local_judge=True)`: writes only to the run's temp directory, no network, a PID namespace that takes down everything the run started, CPU capped) so a mutant stays contained even if the verifier runs it in-process; macOS's sandbox-exec can't reap detached processes, so there mutants need docker, modal or e2b (sandbox providers run mutants only where the whole sandbox is offline: docker, modal, e2b); `--approve --by NAME` (which needs a rerun and every requested check passed) runs QA on a private copy of a self-contained folder and writes a read-only `approval.json` (report, file hashes, version, time, lint warnings marked advisory) that `check` flags once the files change. Reports list skipped checks under `not_run`, and approvals name them. `deepgym_solution` gains `apply_patch` (refusing file creation, deletion, renames and copies) and `load_files` (a normal import of the patched tree) for repo-patch verifiers, and raises `LaunchError` when the machine can't start the solution, so verifiers end the run as an error instead of scoring 0. `empty_fails` and `stub_fails` are now `inconclusive`, not passed, when the verifier errors instead of scoring, `hack_resistant` is skipped, not passed, when the verifier rejects every attack as a malformed answer, `deterministic` is skipped after a single reference run, `reference_passes` and `deterministic` are inconclusive when a reference run gets no verdict, and so is `difficulty` when a sample gets none. Breaking: `run_qa(target, dg, QAOptions(...))` replaces the `reruns`/`client`/`samples` keywords. diff --git a/deepgym/docs/environments/sessions.md b/deepgym/docs/environments/sessions.md index f9aa8ab..9cedaf5 100644 --- a/deepgym/docs/environments/sessions.md +++ b/deepgym/docs/environments/sessions.md @@ -98,12 +98,22 @@ Docker sessions: Docker runs them with no-new-privileges, which blocks sudo. - A step that fails after its action may have run ends the episode: `step()` raises, the action stays on the trajectory with no step reward (so `step_rewards` is one shorter than `steps`), and `score()` returns the result that failure fixed instead of running - the final verifier. A provider or network failure, or a failed step verifier (bad - output, a timeout, exit 2, or an `error_type` that isn't the model's fault), has no - verdict; a failed step verifier reports `verifier_error`, or `invalid_score` for a NaN - or out-of-range score. Anything else follows from the agent's actions (its sandbox - died, a write failed, its files blocked the step's output) and scores 0 with + the final verifier. Only a failure that is ours for certain has no verdict + (`deepgym.session.failure_result`): a provider, network or auth failure, a step + verifier that couldn't run or failed (bad output, exit 2, an `error_type` that isn't the + model's fault, or a timeout before it read the agent's output), or the provider retiring + the sandbox at the end of its lifetime, unless the agent's own actions took at least half + the session's time. A failed step verifier reports `verifier_error`, or `invalid_score` + for a NaN or out-of-range score. Anything else follows from the agent's actions (its + sandbox died, a write failed, even with a disk or I/O error, since the agent can fill + the disk or plant a link, its files blocked the step's output) and scores 0 with `error_type='policy_sandbox_failure'`. +- A step or final verifier that times out after reading the agent's output (it opened a + file, listed a folder or started a program in the work directory or on a file there, or + started the agent's code) is a judged 0 with `error_type='solution_timeout'`; for a step + verifier that is the step's reward, and the episode goes on. One that is killed, or + gives no verdict, after reading it is a judged 0 with `solution_error` (unless the + agent's code failed to launch), and so is a timeout whose output was lost. - A rejected action (not allowed, or the agent's own files in the way of a `write_file`) raises `ActionError` and the episode goes on. - `run_episode` returns the result a failed step fixed instead of raising. diff --git a/deepgym/docs/training/reward-contract.md b/deepgym/docs/training/reward-contract.md index 8bc2a0c..d9474d9 100644 --- a/deepgym/docs/training/reward-contract.md +++ b/deepgym/docs/training/reward-contract.md @@ -53,11 +53,18 @@ agent's work directory. sandbox that dies on Daytona, or in an E2B or Modal call that raises, since those return nothing the judge printed. -In agent episodes, `deepgym.exceptions.is_infrastructure_failure(exc)` draws the line: a -connection or transport error, a provider call timing out, a provider HTTP 5xx or 429, or -the provider evicting the sandbox is infrastructure. A `SandboxDiedError` (the sandbox -itself stopped) never is. Sessions, verl's multi-turn agent loop and the OpenRLHF endpoint -use this test. +In agent episodes, `deepgym.session.failure_result(exc, sandbox_id)` draws the line once the +agent has acted. Only a failure that is ours for certain has no verdict: what +`deepgym.exceptions.is_infrastructure_failure(exc)` calls infrastructure (a connection or +transport error, a provider call timing out, a provider HTTP 5xx or 429, the provider +refusing our credentials with 401 or 403, or the provider evicting the sandbox), a +verifier that couldn't run or timed out before it read the agent's output, or the +provider retiring the sandbox at the end of its lifetime (`SandboxExpiredError`), unless the +agent's own actions took at least half the session's time. Anything else, a +`SandboxDiedError` or a failed write included, is the agent's 0, and so is a failure the +rule can't read. Sessions, verl's +multi-turn agent loop and Harbor's agent phase use this rule; the OpenRLHF endpoint +grades once more what `is_infrastructure_failure` calls infrastructure. ## What each trainer receives diff --git a/deepgym/docs/training/verl.md b/deepgym/docs/training/verl.md index 19486a1..66cd32f 100644 --- a/deepgym/docs/training/verl.md +++ b/deepgym/docs/training/verl.md @@ -99,17 +99,20 @@ tools: - Written against verl 0.9.1. CI runs the loop against a stand-in for verl's; it was also run on Daytona sandboxes through verl 0.9.1's own `ToolAgentLoop._call_tool`. -Only failures that aren't the policy's doing get no verdict and are kept out of training -the same way (`error` 1.0): a sandbox that fails to start, a provider or network failure -during an action or the final verifier (`deepgym.exceptions.is_infrastructure_failure`), -and a step or final verifier that fails before it runs the agent's code. Everything that -follows from the policy's own actions is scored: +Only failures that are ours for certain get no verdict and are kept out of training the +same way (`error` 1.0): a sandbox that fails to start, a provider, network or auth +failure, a provider lifetime that ran out, and a step or final verifier that couldn't +run or timed out before it read the agent's output (`deepgym.session.failure_result`); +an expiry only if the policy's own actions didn't take most of the time. Everything else +is scored: - An invalid or disallowed action comes back to the model as text, and the rollout goes on. - Any other failed action (a failed write, a sandbox the policy killed) comes back as an error and ends the episode with a 0, `error_type` `policy_sandbox_failure`, as does a final verifier that can't run in a sandbox the policy killed or hung. +- A verifier that times out after reading the agent's output scores 0 with + `solution_timeout`. - A rollout where the model never calls the tool scores 0. This loop also warns when more than 5 samples in a block of 100 are kept out. diff --git a/deepgym/src/deepgym/exceptions.py b/deepgym/src/deepgym/exceptions.py index e216dc1..705ecf8 100644 --- a/deepgym/src/deepgym/exceptions.py +++ b/deepgym/src/deepgym/exceptions.py @@ -74,6 +74,12 @@ def __init__(self, message: str, output: str = '') -> None: self.output = output +class SandboxExpiredError(SandboxError): + """Raise when the provider retired the sandbox at the end of its lifetime, whatever ran + in it. Not the policy's doing unless its own actions ran the clock out (see + :func:`deepgym.session.failure_result`).""" + + class TimeoutError(DeepGymError): """Raise when execution exceeds the configured timeout.""" @@ -112,8 +118,9 @@ def is_infrastructure_failure(exc: BaseException) -> bool: """Return True if *exc* is a failure of the network or the sandbox provider. That is a connection or transport error, a provider call timing out, a provider HTTP - 5xx or 429, or the provider evicting the sandbox, anywhere along the exception's - ``__cause__`` chain. Such a run got no verdict, and trainers keep it out of training. + 5xx or 429, the provider refusing our credentials (401, 403), or the provider evicting + the sandbox, anywhere along the exception's ``__cause__`` chain. Such a run got no + verdict, and trainers keep it out of training. Anything else is the policy's doing and is scored: a command or verifier that ran out of time, a sandbox the provider reports gone or stopped after the policy acted, a @@ -132,7 +139,7 @@ def is_infrastructure_failure(exc: BaseException) -> bool: if isinstance(exc, (ConnectionError, builtins.TimeoutError, *transport)): return True status = _status_code(exc) - if status is not None and (status >= 500 or status == 429): + if status is not None and (status >= 500 or status in (401, 403, 429)): return True exc = exc.__cause__ return False diff --git a/deepgym/src/deepgym/harbor/agent.py b/deepgym/src/deepgym/harbor/agent.py index 7407416..a2645d4 100644 --- a/deepgym/src/deepgym/harbor/agent.py +++ b/deepgym/src/deepgym/harbor/agent.py @@ -3,12 +3,14 @@ from __future__ import annotations import math +import time from typing import NamedTuple -from deepgym.exceptions import DeepGymError, TimeoutError +from deepgym.exceptions import DeepGymError, SandboxExpiredError, TimeoutError from deepgym.harbor.task import HARNESS_PATHS, STAGING_DIR, HarborTask +from deepgym.models import RunResult from deepgym.sandboxes import Sandbox -from deepgym.session import SandboxBackend, check_workdir +from deepgym.session import SandboxBackend, check_workdir, failure_result, ran_out_the_clock DEFAULT_AGENT_TIMEOUT_S = 1800 """Seconds the agent's script may run when task.toml sets no ``[agent].timeout_sec``.""" @@ -30,13 +32,20 @@ class AgentRun(NamedTuple): """Whether the script was killed at the agent timeout.""" -def run_agent(sandbox: Sandbox, task: HarborTask, script: str) -> AgentRun: +def run_agent(sandbox: Sandbox, task: HarborTask, script: str) -> AgentRun | RunResult: """Run *script* with bash in *task*'s work directory, then archive that directory. The script runs as an unprivileged user with no capabilities and no network (:class:`~deepgym.session.SandboxBackend`), and owns the work directory. Its processes are killed before the archive is made. A timeout ends the script, not the run. + Returns: + What the agent left behind; or, once its script has started, the result of any + failure (see :func:`~deepgym.session.failure_result`): no verdict only if it is ours + for certain (or the provider's lifetime ran out before the script took half the + time), else the agent's judged 0 (its sandbox died, its work directory can't be + packed or read back, ...). + Args: sandbox: A fresh sandbox from the task's image; deleted afterwards. task: The task; its workdir, else the image's WORKDIR, is the work directory. @@ -44,19 +53,32 @@ def run_agent(sandbox: Sandbox, task: HarborTask, script: str) -> AgentRun: Raises: DeepGymError: If the image's WORKDIR can't be the agent's (``/``, /root, /tmp ...). - SandboxError: If setting up the sandbox or archiving fails. + SandboxError: If setting up the sandbox or uploading the script fails. """ + opened = time.monotonic() workdir = task.workdir or _image_workdir(sandbox, task.name) backend = SandboxBackend(sandbox, workdir) # deletes the sandbox if setup fails try: sandbox.upload(script.encode('utf-8'), _SCRIPT) timeout = math.ceil(task.agent_timeout_sec or DEFAULT_AGENT_TIMEOUT_S) + started = time.monotonic() + acting_s = None try: - backend.exec(f'bash {_SCRIPT}', timeout) - timed_out = False - except TimeoutError: - timed_out = True - return AgentRun(backend.archive_workdir(), backend.workdir, timed_out) + try: + backend.exec(f'bash {_SCRIPT}', timeout) + timed_out = False + except TimeoutError: + timed_out = True + acting_s = time.monotonic() - started + workspace = backend.archive_workdir() + except SandboxExpiredError as exc: + acting_s = time.monotonic() - started if acting_s is None else acting_s + exhausted = ran_out_the_clock(exc, acting_s, time.monotonic() - opened) + return failure_result(exc, sandbox.id, exhausted=exhausted) + except Exception as exc: + # The script has started: only a failure that is ours for certain has no verdict. + return failure_result(exc, sandbox.id) + return AgentRun(workspace, backend.workdir, timed_out) finally: sandbox.delete() # Never raises; a failed delete is logged, not traded for the run. diff --git a/deepgym/src/deepgym/harbor/environment.py b/deepgym/src/deepgym/harbor/environment.py index 087f5c1..cdb8b9c 100644 --- a/deepgym/src/deepgym/harbor/environment.py +++ b/deepgym/src/deepgym/harbor/environment.py @@ -97,6 +97,8 @@ def run_with_deepgym(self, dg: DeepGym, model_output: str, **kwargs: object) -> start = time.perf_counter() env = self if self.image else self.model_copy(update={'image': self._built_image(dg)}) agent = run_agent(dg.open_sandbox(env), self.harbor_task, model_output) + if isinstance(agent, RunResult): # the agent broke its own sandbox or work directory + return agent grader = dg.open_sandbox(env) result = grade(grader, self.harbor_task, agent.workspace, agent.workdir) return build_run_result( diff --git a/deepgym/src/deepgym/integrations/verl_tool.py b/deepgym/src/deepgym/integrations/verl_tool.py index cf5b923..3d89039 100644 --- a/deepgym/src/deepgym/integrations/verl_tool.py +++ b/deepgym/src/deepgym/integrations/verl_tool.py @@ -14,19 +14,20 @@ the model never got to shape. The tool remembers such a failure instead, and the rollout gets no verdict: like :mod:`deepgym.integrations.verl_mask`, the loop keeps it out of training (see :func:`~deepgym.integrations.verl_mask.exclude_unscored`) and logs -``error`` 1.0 in ``reward_extra_info``. Only failures that aren't the policy's doing are -kept out (see :func:`~deepgym.exceptions.is_infrastructure_failure`): a sandbox that -fails to start, a provider or network failure during an action or the final verifier, -and a step or final verifier that fails before it runs the agent's code, as the session -reports. - -Everything that follows from the policy's own actions is scored. An invalid or -disallowed action comes back to the model as text. Any other failure of an action (a -failed write, a sandbox the policy killed) comes back as tool-error text and ends the -episode (see :meth:`deepgym.session.Session.step`) with a 0, ``error_type`` -``policy_sandbox_failure``, as does a final verifier that can't run because of what the -policy did (the sandbox died or hung). A rollout where the model never called the tool -scores 0.0. +``error`` 1.0 in ``reward_extra_info``. The session's rule decides which failures those +are (see :func:`~deepgym.session.failure_result`): a sandbox that fails to start, and +failures of an action, the step verifier or the final verifier that are ours for certain +(a provider, network or auth failure, a provider lifetime that ran out, a verifier that +couldn't run or timed out before it read the policy's output), unless the policy used +the sandbox up (filled its disk, or stalled until its lifetime ran out). + +Everything else is scored. An invalid or disallowed action comes back to the model as +text. Any other failed action, or failed step (a sandbox the policy killed, a write that +failed in it), comes back as tool-error text and ends the episode (see +:meth:`deepgym.session.Session.step`) with a 0, ``error_type`` ``policy_sandbox_failure``, +as does a sandbox that dies under the final verifier; a verifier that times out after +reading the policy's output scores 0 with ``solution_timeout``. A rollout where the model +never called the tool scores 0.0. The share of rollouts kept out is watched (:class:`~deepgym.integrations.reward.MaskedRate`). Agent loop config (``actor_rollout_ref.rollout.agent.agent_loop_config_path``):: @@ -70,19 +71,13 @@ from verl.tools.base_tool import BaseTool, OpenAIFunctionToolSchema, ToolResponse from deepgym.core import DeepGym -from deepgym.exceptions import ( - ActionError, - DeepGymError, - SandboxError, - VerifierError, - is_infrastructure_failure, -) +from deepgym.exceptions import ActionError, DeepGymError, SandboxError from deepgym.integrations.reward import MaskedRate from deepgym.integrations.verl_mask import exclude_unscored from deepgym.models import Action, ActionType, MultiTurnEnvironment, Observation, RunResult from deepgym.registry import load_environment from deepgym.sandbox import error_result -from deepgym.session import Session +from deepgym.session import Session, failure_result logger = logging.getLogger(__name__) @@ -176,11 +171,11 @@ async def execute( Raises: Exception: Whatever starting the sandbox, or applying a valid action, raised. - verl shows it to the model as tool-error text. A failed start, or a provider - or network failure (see :func:`~deepgym.exceptions.is_infrastructure_failure`), - marks the episode failed: it gets no verdict and takes no more actions. Any - other failure follows from the policy's actions: the session ends the - episode there, and it scores 0. + verl shows it to the model as tool-error text. A failed start, or a failure + the policy isn't to blame for (see :func:`~deepgym.session.failure_result`), + marks the episode failed: it gets no verdict and takes no more actions. The + session ends the episode on any failure, and one the policy is to blame for + scores 0. """ episode = self._episode_of[instance_id] async with self._locks.setdefault(episode, asyncio.Lock()): @@ -205,7 +200,9 @@ async def execute( except ActionError as exc: return ToolResponse(text=str(exc)), 0.0, {} except Exception as exc: - if is_infrastructure_failure(exc): + # A failed step ends the episode and fixes its result (see Session.step). + fixed = session.failed_result + if fixed is not None and not fixed.has_verdict: self._failed[episode] = _no_verdict(exc) raise return _to_response(obs), reward, {'done': done} @@ -218,12 +215,11 @@ async def finish(self, episode: str, *, score: bool) -> RunResult | None: """Score (if asked) and delete the episode's sandbox. Returns: - The final verifier's result; a result without a verdict if the sandbox failed - to start, a provider or network failure hit an action or the verifier, or the - verifier failed (``VerifierError``); a 0 with ``error_type`` - ``policy_sandbox_failure`` if the verifier couldn't run for any other reason, - which follows from the policy's actions (a dead or hung sandbox). None if the - episode never started (the model made no tool call) or *score* is False. + The final verifier's result (a 0 with ``policy_sandbox_failure`` if its sandbox + died, see :meth:`~deepgym.session.Session.score`); the result a failed step + fixed; a result without a verdict if the sandbox failed to start, or the + verifier couldn't give one (see :func:`~deepgym.session.failure_result`). None + if the episode never started (the model made no tool call) or *score* is False. """ self._specs.pop(episode, None) self._locks.pop(episode, None) @@ -237,12 +233,8 @@ async def finish(self, episode: str, *, score: bool) -> RunResult | None: return failed return await asyncio.to_thread(session.score) except Exception as exc: - if is_infrastructure_failure(exc) or isinstance(exc, VerifierError): - return _no_verdict(exc) - logger.debug('DeepGym episode broke its sandbox, scored 0: %s', exc) - return error_result(exc, session.sandbox_id).model_copy( - update={'error_type': 'policy_sandbox_failure'} - ) + result = failure_result(exc, session.sandbox_id) + return result if result.has_verdict else _no_verdict(exc) finally: if session is not None: try: diff --git a/deepgym/src/deepgym/sandboxes/cli.py b/deepgym/src/deepgym/sandboxes/cli.py index 9ff2d0e..8eb619f 100644 --- a/deepgym/src/deepgym/sandboxes/cli.py +++ b/deepgym/src/deepgym/sandboxes/cli.py @@ -203,7 +203,8 @@ def upload(self, content: bytes, path: str) -> None: """Write *content* to *path* inside the container, creating parent directories. Raises: - SandboxError: If the write fails. + SandboxDiedError: If the container has stopped. + SandboxError: If the write fails any other way. """ script = f'PATH={TRUSTED_PATH}; mkdir -p "$(dirname "$1")" && cat > "$1"' try: @@ -214,7 +215,8 @@ def upload(self, content: bytes, path: str) -> None: except subprocess.TimeoutExpired as exc: raise SandboxError(f'Upload of {path} timed out') from exc if done.returncode != 0: - raise SandboxError(f'Failed to upload {path}: {done.stderr.decode()[-500:]}') + message = f'Failed to upload {path}: {done.stderr.decode()[-500:]}' + raise SandboxDiedError(message) if self._gone() else SandboxError(message) def delete(self) -> bool: """Stop and remove the container; never raises, and a failure is logged. diff --git a/deepgym/src/deepgym/sandboxes/daytona/sandbox.py b/deepgym/src/deepgym/sandboxes/daytona/sandbox.py index c4f45b9..75708f7 100644 --- a/deepgym/src/deepgym/sandboxes/daytona/sandbox.py +++ b/deepgym/src/deepgym/sandboxes/daytona/sandbox.py @@ -12,6 +12,7 @@ import posixpath import re import shlex +import time from collections.abc import Mapping from types import ModuleType from typing import TYPE_CHECKING @@ -20,6 +21,7 @@ from deepgym.exceptions import SandboxDiedError, SandboxError from deepgym.models import Environment, MultiTurnEnvironment from deepgym.sandboxes.protocol import ( + TIMEOUT_EXIT_CODES, TRUSTED_PATH, ExecResult, SandboxImage, @@ -37,6 +39,9 @@ logger = logging.getLogger(__name__) +# How much longer than the command's own deadline Daytona waits for it to end. +_WAIT_GRACE_S = 15 + def _sdk() -> ModuleType: """Import the daytona SDK. @@ -239,22 +244,32 @@ def exec( ) -> ExecResult: """Run *command* with sh as root. + The deadline is enforced in the sandbox by coreutils ``timeout``, as on E2B, so a + command that outruns it still returns what it printed (Daytona's own timeout + returns nothing); Daytona waits a little longer. + Raises: SandboxTimeoutError: If the command outruns *timeout* seconds. SandboxDiedError: If the sandbox stopped during the command. """ + if timeout: + command = f'/usr/bin/timeout --kill-after=2 {timeout} /bin/sh -c {shlex.quote(command)}' + start = time.monotonic() try: response = self._sandbox.process.exec( f'{self._sudo}/bin/sh -c {shlex.quote(command)}', cwd=cwd, env=dict(env) if env else None, - timeout=timeout, + timeout=timeout + _WAIT_GRACE_S if timeout else None, ) except Exception as exc: if is_timeout(exc): raise SandboxTimeoutError(f'command exceeded {timeout}s') from exc self._raise_if_stopped(exc) raise + if timeout and response.exit_code in TIMEOUT_EXIT_CODES: + if time.monotonic() - start >= timeout: + raise SandboxTimeoutError(f'command exceeded {timeout}s', response.result) return ExecResult(response.exit_code, response.result) def _raise_if_stopped(self, exc: Exception) -> None: @@ -281,13 +296,17 @@ def upload(self, content: bytes, path: str) -> None: """Write *content* to *path* as root, creating parent directories. Raises: + SandboxDiedError: If the sandbox has stopped. SandboxError: If root can't move the staged file into place. """ + staged = f'{self._stage}/{uuid4().hex}' if self._stage else path + try: + self._sandbox.fs.upload_file(content, staged) + except Exception as exc: + self._raise_if_stopped(exc) + raise if not self._stage: - self._sandbox.fs.upload_file(content, path) return - staged = f'{self._stage}/{uuid4().hex}' - self._sandbox.fs.upload_file(content, staged) src, dst = shlex.quote(staged), shlex.quote(path) moved = self.exec( f'PATH={TRUSTED_PATH}; mkdir -p {shlex.quote(posixpath.dirname(path))} && ' diff --git a/deepgym/src/deepgym/sandboxes/e2b/sandbox.py b/deepgym/src/deepgym/sandboxes/e2b/sandbox.py index 7997bfd..e6b60e5 100644 --- a/deepgym/src/deepgym/sandboxes/e2b/sandbox.py +++ b/deepgym/src/deepgym/sandboxes/e2b/sandbox.py @@ -19,12 +19,13 @@ ) from e2b import Sandbox as E2BClient -from deepgym.exceptions import SandboxDiedError, SandboxError +from deepgym.exceptions import SandboxError from deepgym.sandboxes.protocol import ( TIMEOUT_EXIT_CODES, ExecResult, SandboxImage, SandboxTimeoutError, + stopped_error, ) logger = logging.getLogger(__name__) @@ -42,6 +43,7 @@ class E2BSandbox: def __init__(self, template: str | None = None) -> None: self._template = template + self._expires_at = time.monotonic() + LIFETIME_S self._sandbox = E2BClient.create( template=template, timeout=LIFETIME_S, allow_internet_access=False ) @@ -69,6 +71,8 @@ def exec( Raises: SandboxTimeoutError: If the command outruns *timeout* seconds. SandboxDiedError: If the sandbox stopped running during the command. + SandboxExpiredError: If E2B retired it at the end of its lifetime (see + :func:`~deepgym.sandboxes.protocol.stopped_error`). SandboxError: If E2B can't reach the sandbox. """ if timeout: @@ -87,7 +91,7 @@ def exec( result = exc # Both subclass TimeoutException, but mean the sandbox stopped or can't be reached. except SandboxNotRunningException as exc: - raise SandboxDiedError(f'E2B sandbox {self.id} stopped running: {exc}') from exc + raise self._stopped(exc) from exc except SandboxUnreachableException as exc: raise SandboxError(f'E2B sandbox {self.id} is not reachable: {exc}') from exc except TimeoutException as exc: @@ -99,8 +103,20 @@ def exec( return ExecResult(result.exit_code, result.stdout + result.stderr) def upload(self, content: bytes, path: str) -> None: - """Write *content* to *path* as root (E2B creates parent directories).""" - self._sandbox.files.write(path, content, user='root') + """Write *content* to *path* as root (E2B creates parent directories). + + Raises: + SandboxDiedError: If the sandbox has stopped running. + SandboxExpiredError: If E2B retired it at the end of its lifetime. + """ + try: + self._sandbox.files.write(path, content, user='root') + except SandboxNotRunningException as exc: + raise self._stopped(exc) from exc + + def _stopped(self, exc: SandboxNotRunningException) -> SandboxError: + """Return what E2B reporting the sandbox stopped (*exc*) means.""" + return stopped_error(f'E2B sandbox {self.id} stopped running: {exc}', self._expires_at) def delete(self) -> bool: """Kill the sandbox; errors are logged. Return False if E2B reported a failure. diff --git a/deepgym/src/deepgym/sandboxes/modal/sandbox.py b/deepgym/src/deepgym/sandboxes/modal/sandbox.py index 21c53b6..248eb8f 100644 --- a/deepgym/src/deepgym/sandboxes/modal/sandbox.py +++ b/deepgym/src/deepgym/sandboxes/modal/sandbox.py @@ -13,12 +13,12 @@ import modal -from deepgym.exceptions import SandboxDiedError from deepgym.sandboxes.protocol import ( TRUSTED_PATH, ExecResult, SandboxImage, SandboxTimeoutError, + stopped_error, ) logger = logging.getLogger(__name__) @@ -36,6 +36,7 @@ class ModalSandbox: def __init__(self, image: str | None = None) -> None: self._image = image app = modal.App.lookup(APP_NAME, create_if_missing=True) + self._expires_at = time.monotonic() + LIFETIME_S self._sandbox = modal.Sandbox.create( 'sleep', 'infinity', app=app, @@ -70,6 +71,8 @@ def exec( Raises: SandboxTimeoutError: If the command outruns *timeout* seconds. SandboxDiedError: If Modal terminated the sandbox during the command. + SandboxExpiredError: If Modal retired it at the end of its lifetime (see + :func:`~deepgym.sandboxes.protocol.stopped_error`). """ start = time.monotonic() try: @@ -82,11 +85,12 @@ def exec( except modal.exception.Error as exc: # Modal words a stopped sandbox several ways (terminated, not found); poll() tells. if isinstance(exc, modal.exception.SandboxTerminatedError) or self._stopped(): - raise SandboxDiedError(f'Modal sandbox {self.id} was terminated: {exc}') from exc + message = f'Modal sandbox {self.id} was terminated: {exc}' + raise stopped_error(message, self._expires_at) from exc raise if exit_code != 0 and self._stopped(): message = f'Modal sandbox {self.id} was terminated: {output[-500:]}' - raise SandboxDiedError(message, output) + raise stopped_error(message, self._expires_at, output) # Modal kills an exec at its timeout and reports only a failed exit. if timeout and exit_code != 0 and time.monotonic() - start >= timeout: raise SandboxTimeoutError(f'command exceeded {timeout}s', output) diff --git a/deepgym/src/deepgym/sandboxes/protocol.py b/deepgym/src/deepgym/sandboxes/protocol.py index d339b76..4b24103 100644 --- a/deepgym/src/deepgym/sandboxes/protocol.py +++ b/deepgym/src/deepgym/sandboxes/protocol.py @@ -17,10 +17,11 @@ from __future__ import annotations +import time from collections.abc import Mapping from typing import NamedTuple, Protocol -from deepgym.exceptions import TimeoutError +from deepgym.exceptions import SandboxDiedError, SandboxError, SandboxExpiredError, TimeoutError TRUSTED_PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' """Root-owned system folders only; no agent work directory may hold or sit in them.""" @@ -62,6 +63,24 @@ def __init__(self, message: str, output: str = '') -> None: self.output = output +def stopped_error(message: str, expires_at: float, output: str = '') -> SandboxError: + """Return the error for a sandbox its provider reports stopped. + + From *expires_at* on (``time.monotonic()``, taken before the sandbox was created, so + never after the provider's own deadline), the provider retired it at the end of its + lifetime, whatever ran in it: :class:`~deepgym.exceptions.SandboxExpiredError`. Before + then something in it stopped it: :class:`SandboxDiedError`. + + Args: + message: What happened. + expires_at: When the provider's lifetime for the sandbox runs out. + output: What the command printed first, for :class:`SandboxDiedError`. + """ + if time.monotonic() >= expires_at: + return SandboxExpiredError(f'{message} (its provider lifetime was up)') + return SandboxDiedError(message, output) + + class Sandbox(Protocol): """A running sandbox, deleted by :meth:`delete`.""" @@ -92,7 +111,11 @@ def exec( ... def upload(self, content: bytes, path: str) -> None: - """Write *content* to the absolute *path* as root, creating parent directories.""" + """Write *content* to the absolute *path* as root, creating parent directories. + + Raises: + SandboxDiedError: If the sandbox itself has stopped. + """ ... def delete(self) -> bool: diff --git a/deepgym/src/deepgym/session.py b/deepgym/src/deepgym/session.py index 0fe2f01..b84b199 100644 --- a/deepgym/src/deepgym/session.py +++ b/deepgym/src/deepgym/session.py @@ -34,6 +34,7 @@ DeepGymError, SandboxDiedError, SandboxError, + SandboxExpiredError, TimeoutError, VerifierError, is_infrastructure_failure, @@ -57,15 +58,25 @@ is_verdict, ) from deepgym.sandbox import ( + ExecutionResult, _extract_json_line, + _judged_zero, build_run_result, check_exit, error_result, parse_verifier_output, + solution_timeout, split_wrapped_output, wrap_stderr, ) -from deepgym.sandboxes import TRUSTED_PATH, ExecResult, Sandbox, SandboxImage, sealed_path +from deepgym.sandboxes import ( + TRUSTED_PATH, + ExecResult, + Sandbox, + SandboxImage, + SandboxTimeoutError, + sealed_path, +) from deepgym.verifier_template import guard_crashes if TYPE_CHECKING: @@ -112,16 +123,21 @@ _DISPLAY = ':0' _STEP_OUTPUT = '_step_output.txt' # errno names for a write the agent's own files block (a folder where the file goes, a file -# where a folder goes, a link or mode it set, a name too long), raised before any byte is -# written. Any other write failure is the sandbox's. Names, as the numbers vary by platform. -_BLOCKED_ERRNOS = ('EACCES', 'EEXIST', 'EISDIR', 'ELOOP', 'ENAMETOOLONG', 'ENOTDIR', 'EPERM') +# where a folder goes, a link or mode it set, a name too long, a FIFO with no reader), raised +# before any byte is written. Any other write failure is the sandbox's. Names, as the +# numbers vary by platform. +_BLOCKED_ERRNOS = ( + 'EACCES', 'EEXIST', 'EISDIR', 'ELOOP', 'ENAMETOOLONG', 'ENOTDIR', 'ENXIO', 'EPERM', +) # fmt: skip +# Opens a write target without blocking on a FIFO the agent left there (ENXIO instead). +_WRITE_FLAGS = os.O_WRONLY | os.O_CREAT | os.O_TRUNC | os.O_NONBLOCK # Run as the session user to write stdin to argv[1]: exits 3 if the agent's files block it, # including a link it planted at the path, whatever the error following that link. _WRITE_AS_AGENT = f""" import errno, os, sys try: os.makedirs(os.path.dirname(sys.argv[1]), exist_ok=True) - fd = os.open(sys.argv[1], os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o666) + fd = os.open(sys.argv[1], os.O_WRONLY | os.O_CREAT | os.O_TRUNC | os.O_NONBLOCK, 0o666) except OSError as exc: print(exc, file=sys.stderr) blocked = exc.errno in {{getattr(errno, n) for n in {_BLOCKED_ERRNOS}}} @@ -221,7 +237,7 @@ def _run(self, command: str, timeout: int, env: dict[str, str] | None) -> Verifi """Run *command* with bash in the temp directory, keeping stdout and stderr apart. Raises: - TimeoutError: If the command exceeds *timeout* seconds. + SandboxTimeoutError: If the command exceeds *timeout* seconds, with its stdout. """ proc_env = {'PATH': '/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin', **(env or {})} # Own process group, so background children die with the command or the session. @@ -230,6 +246,7 @@ def _run(self, command: str, timeout: int, env: dict[str, str] | None) -> Verifi stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + errors='replace', cwd=self._dir, env=proc_env, start_new_session=True, @@ -239,8 +256,8 @@ def _run(self, command: str, timeout: int, env: dict[str, str] | None) -> Verifi stdout, stderr = proc.communicate(timeout=timeout) except subprocess.TimeoutExpired as exc: _kill_group(proc.pid) - proc.communicate() - raise TimeoutError(f'Command exceeded {timeout}s timeout') from exc + stdout, _ = proc.communicate() + raise SandboxTimeoutError(f'Command exceeded {timeout}s timeout', stdout) from exc return VerifierOutput(proc.returncode, stdout, stderr) def write_file(self, path: str, content: bytes) -> None: @@ -253,7 +270,7 @@ def write_file(self, path: str, content: bytes) -> None: target = self._dir / path try: target.parent.mkdir(parents=True, exist_ok=True) - fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o666) + fd = os.open(target, _WRITE_FLAGS, 0o666) except OSError as exc: if errno.errorcode.get(exc.errno) in _BLOCKED_ERRNOS or target.is_symlink(): raise ActionError(f'Failed to write {path}: {exc}') from exc @@ -580,7 +597,9 @@ def archive_workdir(self) -> bytes: Symlinks inside are stored as links, never followed. Raises: - SandboxError: If packing fails. + ActionError: If the agent's work directory can't be packed (it was moved or + replaced by a link, is over the size cap, or the disk is full): its doing. + SandboxError: If reading the archive back fails. """ staged = f'{_STAGING_DIR}/{uuid4().hex}.tgz' quoted = shlex.quote(self._workdir) @@ -600,7 +619,7 @@ def archive_workdir(self) -> bytes: timeout=_TRANSFER_TIMEOUT_S, ) if packed.exit_code != 0: - raise SandboxError(f'Failed to archive {self._workdir}: {packed.output[-500:]}') + raise ActionError(f'Failed to archive {self._workdir}: {packed.output[-500:]}') encoded = self._sandbox.exec( trusted(f'base64 {staged}; status=$?; rm -f {staged}; exit $status'), env=self._clear, @@ -677,14 +696,23 @@ def verify(self, code: str, timeout: int, env: dict[str, str] | None = None) -> Raises: TimeoutError: If the verifier exceeds *timeout* seconds. + SandboxError: If the sandbox stopped (see :class:`~deepgym.sandboxes.Sandbox`). + VerifierError: If the provider fails to take the verifier's code, which is ours + alone. A failed launch isn't: it runs in the agent's work directory. """ script = f'{_VERIFIER_DIR}/{uuid4().hex}.py' - self._sandbox.upload(code.encode('utf-8'), script) command, marker = wrap_stderr( f'{self._offline}{self._as_verifier} {self.python} -I {script}' ) # The deadline is set by the sandbox's own clock, as the judge's is. deadline = f'export DEEPGYM_DEADLINE=$(( $(date +%s) + {timeout} ))' + try: + self._sandbox.upload(code.encode('utf-8'), script) + except (SandboxDiedError, SandboxExpiredError): + raise + except Exception as exc: + # Our code to our own folder: the provider refusing it is never the agent's. + raise VerifierError(f'Could not upload the verifier: {exc}') from exc done = self._sandbox.exec( trusted( f'{deadline}; {command}; status=$?; rm -f {script}; exit $status', @@ -859,6 +887,9 @@ def __init__(self, env: MultiTurnEnvironment, backend: SessionBackend) -> None: self._step_verdict: dict[str, object] | None = None # The episode's result, once a failed step fixed it (see step()). self._failed_result: RunResult | None = None + # The session's start, and the seconds spent applying the agent's actions since. + self._opened = time.monotonic() + self._acting_s = 0.0 # One sandbox, one actor at a time: concurrent step/score/close calls queue up. self._lock = threading.Lock() @@ -882,6 +913,11 @@ def step_verdict(self) -> dict[str, object] | None: """ return self._step_verdict + @property + def failed_result(self) -> RunResult | None: + """The result a failed step fixed (see :meth:`step`); None while no step failed.""" + return self._failed_result + @property def sandbox_id(self) -> str: """Identifier of the backing sandbox.""" @@ -926,16 +962,18 @@ def step(self, action: Action) -> StepResult: and the step is recorded only once both are in. A step that fails any other way than a rejected action ends the episode: the action may have changed the sandbox, so it stays on the record, with no step reward and no retry. :meth:`score` then reports - no verdict for a network or provider failure or a failed step verifier; anything - else follows from the agent's actions (its sandbox died, its files blocked the - step's output file): a judged 0 (``policy_sandbox_failure``). + what :func:`failure_result` makes of the failure: no verdict only where it is + ours for certain and the agent didn't use the sandbox up, else a judged 0. A step + verifier that times out after reading the agent's output gives the step a judged 0 + (``solution_timeout``) instead, and the episode goes on. Raises: DeepGymError: If the session is not running. ActionError: If the action is not allowed here, the desktop rejects it, or the agent's own files are in the way of a write (the episode goes on, unless they blocked the step's output file). - VerifierError: If the step verifier fails or gives no verdict. + VerifierError: If the step verifier fails, gives no verdict, or times out before + it reads the agent's output. SandboxError: If the sandbox fails to apply the action, take the screenshot or run the step verifier. """ @@ -979,12 +1017,16 @@ def score(self) -> RunResult: An episode that a failed step ended (see :meth:`step`) keeps the result that step fixed, with the failure as its ``error_type``; the final verifier doesn't run. A sandbox that dies under the final verifier is the agent's doing once it has acted - (a judged 0); before any action, it has no verdict. + (a judged 0); before any action, it has no verdict. So is a verifier that times out + after reading the agent's output (``solution_timeout``), and a provider lifetime the + agent's own actions ran out (see :meth:`_ran_out_the_clock`). Raises: - VerifierError: If the verifier output is not valid protocol JSON, or the - verifier was killed after printing it. - TimeoutError: If the verifier exceeds the per-step timeout. + VerifierError: If, before it read the agent's output, the verifier timed out or + its output is not valid protocol JSON or isn't backed by its exit code; or its + code couldn't be uploaded. After, any of these is the agent's judged 0 + (``solution_error``), but for the agent's code failing to launch. + TimeoutError: If the verifier timed out and the provider returned nothing. """ with self._lock: if self._failed_result is not None: @@ -1001,20 +1043,32 @@ def score(self) -> RunResult: error_type='no_verifier', ) start = time.perf_counter() + acted = bool(self._trajectory.steps) try: - exit_code, stdout, stderr = self._backend.verify( - _guarded(self._env.final_verifier_code), self._env.timeout_per_step - ) + (exit_code, stdout, stderr), consulted = self._verify(self._env.final_verifier_code) except SandboxDiedError as exc: - if not self._trajectory.steps: + if not acted: return error_result(exc, self.sandbox_id) - return _failure_result(exc, self.sandbox_id) + return failure_result(exc, self.sandbox_id) + except Exception as exc: + if acted and self._ran_out_the_clock(exc): + return failure_result(exc, self.sandbox_id, exhausted=True) + raise stderr = stderr.strip() - result = parse_verifier_output(stdout, f'exit_code={exit_code} stderr: {stderr!r}') elapsed_ms = (time.perf_counter() - start) * 1000 - return build_run_result( - result, elapsed_ms, self.sandbox_id, stderr=stderr, exit_code=exit_code - ) + try: + verdict = parse_verifier_output(stdout, f'exit_code={exit_code} stderr: {stderr!r}') + result = build_run_result( + verdict, elapsed_ms, self.sandbox_id, stderr=stderr, exit_code=exit_code + ) + except VerifierError as exc: + if not consulted or _launch_failed(stdout): + raise + result = _failed_after_reading(exc, elapsed_ms, self.sandbox_id, stderr) + if consulted and not result.has_verdict and result.error_type != 'sandbox_error': + detail = VerifierError(f'{result.error_type}: {result.output}') + return _failed_after_reading(detail, elapsed_ms, self.sandbox_id, stderr) + return result def close(self) -> None: """Release the backend, waiting for any in-flight step to finish.""" @@ -1037,28 +1091,35 @@ def __exit__(self, *args: object) -> None: ) def _apply(self, action: Action) -> tuple[str, int | None]: - """Execute *action*; return its text output and exit code (None if not a command).""" + """Execute *action*; return its text output and exit code (None if not a command). + + Its time, however it ends, counts as the agent's (see :meth:`_ran_out_the_clock`). + """ timeout = self._env.timeout_per_step + started = time.monotonic() try: - match action.action_type: - case 'code': - exit_code, output = self._run_python('_step.py', action.content) - return output, exit_code - case 'bash': - exit_code, output = self._backend.exec(action.content, timeout) - return output, exit_code - case 'write_file': - self._backend.write_file(str(action.path), action.content.encode('utf-8')) - return f'wrote {action.path}', None - case 'wait': - time.sleep(min(action.amount, timeout)) - return '', None - case 'done' | 'screenshot': - return '', None - except TimeoutError: - return f'[timed out after {timeout}s]', None - self._backend.act(action) - return '', None + try: + match action.action_type: + case 'code': + exit_code, output = self._run_python('_step.py', action.content) + return output, exit_code + case 'bash': + exit_code, output = self._backend.exec(action.content, timeout) + return output, exit_code + case 'write_file': + self._backend.write_file(str(action.path), action.content.encode('utf-8')) + return f'wrote {action.path}', None + case 'wait': + time.sleep(min(action.amount, timeout)) + return '', None + case 'done' | 'screenshot': + return '', None + except TimeoutError: + return f'[timed out after {timeout}s]', None + self._backend.act(action) + return '', None + finally: + self._acting_s += time.monotonic() - started def _run_python(self, filename: str, code: str, timeout: int | None = None) -> ExecResult: self._backend.write_file(filename, code.encode('utf-8')) @@ -1067,12 +1128,13 @@ def _run_python(self, filename: str, code: str, timeout: int | None = None) -> E ) def _end(self, action: Action, observation: Observation | None, exc: BaseException) -> None: - """End the episode after *action*'s step failed with *exc* (see :func:`_failure_result`). + """End the episode after *action*'s step failed with *exc* (see :func:`failure_result`). The action may have changed the sandbox, so it is recorded, with no step reward. *observation* is the one taken after the action, if any. """ - self._failed_result = _failure_result(exc, self.sandbox_id) + exhausted = self._ran_out_the_clock(exc) + self._failed_result = failure_result(exc, self.sandbox_id, exhausted=exhausted) if observation is not None: observation = observation.model_copy(update={'done': True}) self._record(action, observation, reward=None, done=True) @@ -1117,21 +1179,33 @@ def _step_reward(self, output: str) -> float: Raises: InvalidScoreError: If the score is NaN, infinite, or outside [0, 1]. - VerifierError: If the step verifier times out, prints neither a float nor - JSON with a score, doesn't exit cleanly (see :func:`check_exit`), or - reports its own failure (exit 2, or an ``error_type`` that isn't the - model's fault; see :func:`~deepgym.models.is_verdict`). + VerifierError: If, before it read the agent's output, the step verifier times + out, prints neither a float nor JSON with a score, doesn't exit cleanly (see + :func:`check_exit`), or reports its own failure (exit 2, or an + ``error_type`` that isn't the model's fault; see + :func:`~deepgym.models.is_verdict`). After, any of these is the step's judged + 0 (``solution_error``), but for the agent's code failing to launch. """ self._backend.write_file(_STEP_OUTPUT, output.encode('utf-8')) try: - exit_code, stdout, _ = self._backend.verify( - _guarded(self._env.step_verifier_code), - self._env.timeout_per_step, - env={'STEP_OUTPUT_PATH': _STEP_OUTPUT, 'STEP_NUMBER': str(self._step)}, + (exit_code, stdout, _), consulted = self._verify( + self._env.step_verifier_code, + {'STEP_OUTPUT_PATH': _STEP_OUTPUT, 'STEP_NUMBER': str(self._step)}, ) except TimeoutError as exc: raise VerifierError(f'Step verifier timed out at step {self._step}') from exc - stdout = stdout.strip() + try: + return self._step_score(exit_code, stdout.strip()) + except VerifierError as exc: + if not consulted or _launch_failed(stdout): + raise + details = f"The step verifier failed after reading the agent's output: {exc}" + verdict = _judged_zero(details, 'solution_error').verifier_result + self._step_verdict = {**verdict.model_dump(), 'exit_code': 1} + return 0.0 + + def _step_score(self, exit_code: int, stdout: str) -> float: + """Return the score in the step verifier's *stdout* (see :meth:`_step_reward`).""" json_str = _extract_json_line(stdout) try: data = json.loads(json_str) if json_str else {'score': float(stdout)} @@ -1151,6 +1225,53 @@ def _step_reward(self, output: str) -> float: self._step_verdict = {**data, 'exit_code': exit_code} return score + def _verify(self, code: str, env: dict[str, str] | None = None) -> tuple[VerifierOutput, bool]: + """Run session verifier *code* against the sandbox as it stands. + + It gets a fresh ``DEEPGYM_CONSULTED`` mark in its code (see :func:`_guarded`). It + prints the mark plus ``:start`` before any of its code runs, and the mark itself once + the agent's output is in play (it read the agent's work directory, ran a program there + or on the agent's files, or started the agent's code, see + :func:`~deepgym.verifier_template.guard_crashes`). Both come as ``\\n\\n`` and + are cut from its stdout exactly, so what it printed reads as if they weren't there. + After the mark, a timeout is the agent's judged 0 (``solution_timeout``, as in a single + run, see :func:`~deepgym.sandbox.solution_timeout`), and so is the verifier being + killed (``solution_error``). So is a timeout whose output the backend lost (no start + line), since nothing then shows the verifier hadn't read the agent's output. A judged + 0 comes back as if the verifier had printed it. + + Returns: + What the verifier printed, and whether it read the agent's output. + + Raises: + VerifierError: If the verifier timed out before the mark. + """ + mark = uuid4().hex + consulted, started = f'\n{mark}\n', f'\n{mark}:start\n' + timeout = self._env.timeout_per_step + try: + exit_code, stdout, stderr = self._backend.verify(_guarded(code, mark), timeout, env) + except SandboxTimeoutError as exc: + if consulted in exc.output: + return _reported(solution_timeout(timeout)), True + if started not in exc.output: + details = f'The verifier timed out and nothing it printed came back: {exc}' + return _reported(_judged_zero(details, 'policy_sandbox_failure')), True + raise VerifierError( + f"The verifier timed out before it read the agent's output: {exc}" + ) from exc + read = consulted in stdout + if read and not 0 <= exit_code < 128: + details = f"The verifier was killed (exit {exit_code}) handling the agent's output" + return _reported(_judged_zero(details, 'solution_error')), True + stdout = stdout.replace(consulted, '').replace(started, '') + return VerifierOutput(exit_code, stdout, stderr), read + + def _ran_out_the_clock(self, exc: BaseException) -> bool: + """Whether *exc* is a provider lifetime the agent's actions ran out (see + :func:`ran_out_the_clock`).""" + return ran_out_the_clock(exc, self._acting_s, time.monotonic() - self._opened) + def run_episode( session: Session, agent: Callable[[Observation], Action] @@ -1176,18 +1297,48 @@ def run_episode( return session.trajectory, session.score() -def _failure_result(exc: BaseException, sandbox_id: str) -> RunResult: +def ran_out_the_clock(exc: BaseException, acting_s: float, elapsed_s: float) -> bool: + """Whether *exc* is the provider retiring the sandbox at the end of its lifetime after + the agent's own actions took at least half of the *elapsed_s* seconds (*acting_s*): then + the agent ran the clock out (stalling with sleeps or waits, say), not the host. + + ponytail: half is a line, not a measure of fault; an agent that stalls for less needs + the host to have spent as long (inference, say) for the lifetime to run out. + """ + return isinstance(exc, SandboxExpiredError) and 2 * acting_s >= elapsed_s + + +def failure_result(exc: BaseException, sandbox_id: str, *, exhausted: bool = False) -> RunResult: """Return the result of an episode that *exc* ended once the agent had acted. - A network or provider failure (see :func:`~deepgym.exceptions.is_infrastructure_failure`), - a step verifier that failed, or an interrupt has no verdict. Anything else follows from - the agent's actions (its sandbox died or hung, a write failed, its files blocked a - harness write): a judged 0 (``policy_sandbox_failure``), or breaking the sandbox would - dodge the score. + The rule for sessions, shared with :mod:`deepgym.integrations.verl_tool` and Harbor's + agent phase. Only a failure that is ours for certain has no verdict: + + * a network or provider failure, or the provider refusing our credentials (see + :func:`~deepgym.exceptions.is_infrastructure_failure`); + * a verifier that couldn't run (its upload or launch failed), failed, or timed out + before it read the agent's output (:class:`~deepgym.exceptions.VerifierError`; after, + the verifier reports the agent's 0 itself, see :meth:`Session._verify`); + * the provider retiring the sandbox at the end of its lifetime + (:class:`~deepgym.exceptions.SandboxExpiredError`), unless *exhausted*: the agent's + own actions ran the clock out (see :meth:`Session._ran_out_the_clock`); + * an interrupt. + + Anything else follows from the agent's actions (its sandbox died, its files blocked a + harness write, a write failed on a disk it fills or a path it controls, a command that + hung): a judged 0 (``policy_sandbox_failure``), or breaking the sandbox would dodge the + score. So is a failure the rule can't read. """ - result = error_result(exc, sandbox_id) - unjudged = isinstance(exc, VerifierError) or not isinstance(exc, Exception) - if unjudged or is_infrastructure_failure(exc): + if not isinstance(exc, Exception): + return error_result(exc, sandbox_id) + try: + ours = isinstance(exc, VerifierError | SandboxExpiredError) and not exhausted + ours = ours or is_infrastructure_failure(exc) + result = error_result(exc, sandbox_id) + except Exception: # an exception that breaks the rule is never a reason to mask it + ours = False + result = error_result(SandboxError(type(exc).__name__), sandbox_id) + if ours: return result return result.model_copy(update={'error_type': 'policy_sandbox_failure'}) @@ -1216,10 +1367,44 @@ def _checked(what: str, result: ExecResult) -> str: """ -def _guarded(code: str) -> str: - """Return session verifier *code* with deepgym_solution on its path, exiting 2 on a crash - (see :func:`~deepgym.verifier_template.guard_crashes`) so it never counts as a failed run.""" - return guard_crashes(code, _RUNTIME_ON_PATH) +def _guarded(code: str, mark: str) -> str: + """Return session verifier *code* with deepgym_solution on its path and *mark* as its + DEEPGYM_CONSULTED, exiting 2 on a crash (see + :func:`~deepgym.verifier_template.guard_crashes`) so it never counts as a failed run. + + The mark goes in the code, which only the verifier user reads, not in a variable on a + command line any process in the sandbox can list. + """ + prelude = ( + f'{_RUNTIME_ON_PATH}_dg_os.environ["DEEPGYM_CONSULTED"] = {mark!r}\n' + f'_dg_os.write(1, b"\\n{mark}:start\\n")\n' + ) + return guard_crashes(code, prelude) + + +def _launch_failed(stdout: str) -> bool: + """Whether a verifier's last protocol line reports that the agent's code couldn't be + launched (the crash guard's ``sandbox_error``): the machine's failure, even after the + verifier read the agent's output.""" + try: + report = json.loads(_extract_json_line(stdout) or 'null') + except ValueError: + return False + return isinstance(report, dict) and report.get('error_type') == 'sandbox_error' + + +def _failed_after_reading( + exc: BaseException, elapsed_ms: float, sandbox_id: str, stderr: str +) -> RunResult: + """Return the judged 0 of a verifier that failed after reading the agent's output.""" + details = f"The verifier failed after reading the agent's output: {exc}" + verdict = _judged_zero(details, 'solution_error').verifier_result + return build_run_result(verdict, elapsed_ms, sandbox_id, stderr=stderr, exit_code=1) + + +def _reported(executed: ExecutionResult) -> VerifierOutput: + """Return *executed*, a judged 0, as the verifier would have printed it.""" + return VerifierOutput(executed.exit_code, executed.verifier_result.model_dump_json(), '') def _inline_python(python: str, code: str) -> str: diff --git a/deepgym/src/deepgym/solution_runtime.py b/deepgym/src/deepgym/solution_runtime.py index afbf5bd..39613fc 100644 --- a/deepgym/src/deepgym/solution_runtime.py +++ b/deepgym/src/deepgym/solution_runtime.py @@ -481,7 +481,10 @@ def _announce() -> None: mark = os.environ.get('DEEPGYM_CONSULTED') if mark and not _announced: _announced = True - print(f'\n{mark}', flush=True) + # Straight to the stdout file, past any capture of sys.stdout, after what was printed. + with contextlib.suppress(AttributeError, OSError, ValueError): + sys.__stdout__.flush() + os.write(1, f'\n{mark}\n'.encode()) def consulted() -> bool: @@ -680,8 +683,10 @@ def run_solution( ) try: _watch_group(proc.pid) - release() + # The mark goes out first: a solution that dies or hangs the moment it is + # released must not beat it and pass for one that never started. _announce() + release() except BaseException: _kill(proc) # held until released, so no solution code ran raise diff --git a/deepgym/src/deepgym/verifier_template.py b/deepgym/src/deepgym/verifier_template.py index 001968c..286014f 100644 --- a/deepgym/src/deepgym/verifier_template.py +++ b/deepgym/src/deepgym/verifier_template.py @@ -181,7 +181,9 @@ def _run_verifier(solution_path, test_cases_path=None): import traceback as _dg_traceback # Where the policy's output lives (the solution, a session's work directory): opening a -# file there counts as consulting it, as starting the solution through deepgym_solution does. +# file there, listing a folder there or starting a program there (a session verifier's own +# working folder is the agent's) counts as consulting it, as starting the solution through +# deepgym_solution does. _dg_policy = tuple( _dg_os.path.join(path, "") for entry in _dg_os.environ.get("DEEPGYM_POLICY_DIR", "").split(_dg_os.pathsep) @@ -189,25 +191,71 @@ def _run_verifier(solution_path, test_cases_path=None): for path in {entry, _dg_os.path.realpath(entry)} ) _dg_touched = False +# Audit events that read the path in their first argument (None: the working folder). +_dg_reads = {"open", "os.listdir", "os.scandir", "glob.glob", "glob.glob/2"} +# Audit events that start a program, which works in the working folder (subprocess.Popen's +# cwd argument when given), with where its program and argument list are (os.system has none). +_dg_starts = { + "subprocess.Popen": (0, 1), + "os.system": (), + "os.posix_spawn": (0, 1), + "os.exec": (0, 1), + "os.spawn": (1, 2), +} def _dg_announce(): - """Tell the host on stdout that the policy's output is in play (DEEPGYM_CONSULTED).""" + """Tell the host on stdout that the policy's output is in play (DEEPGYM_CONSULTED). + + Written to the stdout file itself, so a verifier that captures sys.stdout can't hide it, + on a line of its own after whatever the verifier printed before. + """ mark = _dg_os.environ.get("DEEPGYM_CONSULTED") if mark: - print("\\n" + mark, flush=True) + try: + if _dg_sys.__stdout__ is not None: + _dg_sys.__stdout__.flush() + _dg_os.write(1, ("\\n" + mark + "\\n").encode()) + except (OSError, ValueError): + pass # an audit hook that raises would fail the verifier's own call + + +def _dg_inside(target): + """Whether *target* (a path; None for the working folder) is in the policy's folders.""" + try: + path = _dg_os.path.abspath(_dg_os.fsdecode("." if target is None else target)) + except (TypeError, ValueError): + return False # a file descriptor, say + return _dg_os.path.join(path, "").startswith(_dg_policy) def _dg_audit(event, args): - """Count opening one of the policy's files as consulting it.""" + """Count reading one of the policy's files or folders, or starting a program in one or + on one (an absolute path of the policy's among its program and arguments), as + consulting it.""" global _dg_touched - if _dg_touched or event != "open" or not _dg_policy: + if _dg_touched or not _dg_policy: return - try: - path = _dg_os.path.join(_dg_os.path.abspath(_dg_os.fsdecode(args[0])), "") - except (TypeError, ValueError): - return # a file descriptor, say - if path.startswith(_dg_policy): + if event in _dg_reads: + hit = _dg_inside(args[2] if event == "glob.glob/2" and args[2] is not None else args[0]) + elif event in _dg_starts: + named = [] + for index in _dg_starts[event]: + value = args[index] + named += value if isinstance(value, (list, tuple)) else [value] + absolute = [] + for arg in named: + try: + text = _dg_os.fsdecode(arg) # a str, bytes or path object + except TypeError: + continue + if text.startswith("/"): + absolute.append(text) + cwd = args[2] if event == "subprocess.Popen" else None + hit = _dg_inside(cwd) or any(_dg_inside(arg) for arg in absolute) + else: + return + if hit: _dg_touched = True _dg_announce() diff --git a/deepgym/tests/test_exceptions.py b/deepgym/tests/test_exceptions.py index 0a75d8e..5e7c515 100644 --- a/deepgym/tests/test_exceptions.py +++ b/deepgym/tests/test_exceptions.py @@ -47,6 +47,8 @@ def __init__(self, status_code: int) -> None: _status(502), _SdkError(503), _SdkError(429), + _status(401), # our credentials + _SdkError(403), # DeepGym wraps provider errors; the cause decides. _caused_by(SandboxError('click failed'), ConnectionRefusedError()), ], @@ -58,6 +60,8 @@ def __init__(self, status_code: int) -> None: '502', 'sdk_503', 'rate_limit', + 'unauthorized', + 'forbidden', 'wrapped', ], ) diff --git a/deepgym/tests/test_harbor_isolation.py b/deepgym/tests/test_harbor_isolation.py index b81607c..fd8acde 100644 --- a/deepgym/tests/test_harbor_isolation.py +++ b/deepgym/tests/test_harbor_isolation.py @@ -11,12 +11,19 @@ import shutil import subprocess import tarfile +import time from pathlib import Path import pytest from deepgym.benchmark_envs import TerminalBenchEnvironment -from deepgym.exceptions import DeepGymError, SandboxError, VerifierError +from deepgym.exceptions import ( + DeepGymError, + SandboxDiedError, + SandboxError, + SandboxExpiredError, + VerifierError, +) from deepgym.harbor import HarborEnvironment, HarborTask from deepgym.harbor.agent import run_agent from deepgym.harbor.grading import grade @@ -103,10 +110,43 @@ def exec_(command: str, **kwargs: object) -> object: assert run_agent(box, HarborTask.from_dir(HELLO), 'sleep 999').timed_out -def test_a_corrupted_archive_is_a_sandbox_error() -> None: +@pytest.mark.parametrize( + 'archive', ['unreadable', 'refused', 'host_error', 'provider_outage', 'not_started'] +) +def test_after_the_script_starts_only_a_failure_that_is_ours_for_certain_is_masked( + archive: str, +) -> None: + """Fail closed: the agent can make its archive fail (fill the disk, unreadable files), + so any failure once its script started is its 0, unless it is certainly ours. A + failure before the script starts (its upload) stays ours.""" box = _box(('base64 /opt', 'H4sI\nnot base64: warning from stderr\n')) - with pytest.raises(SandboxError, match='read the archive'): - run_agent(box, HarborTask.from_dir(HELLO), '') + real_exec = box.exec + errors = { + 'refused': SandboxError('the provider refused the command'), + 'host_error': OSError(5, 'Input/output error'), + 'provider_outage': ConnectionError('provider unreachable'), + } + + def exec_(command: str, **kwargs: object) -> object: + if '-czf' in command and archive in errors: + raise errors[archive] + return real_exec(command, **kwargs) + + def upload(content: bytes, path: str) -> None: + raise SandboxError('script upload failed') + + box.exec = exec_ # type: ignore[method-assign] + if archive == 'not_started': + box.upload = upload # type: ignore[method-assign] + with pytest.raises(SandboxError, match='script upload failed'): + run_agent(box, HarborTask.from_dir(HELLO), '') + assert box.deleted + return + result = run_agent(box, HarborTask.from_dir(HELLO), '') + + ours = archive == 'provider_outage' + assert (result.score, result.has_verdict) == (0.0, not ours) + assert result.error_type == ('unknown_error' if ours else 'policy_sandbox_failure') assert box.deleted @@ -122,16 +162,33 @@ def test_a_failed_delete_costs_neither_the_run_nor_the_grade() -> None: assert agent_box.deleted and grader_box.deleted -def test_agent_sandbox_is_deleted_when_archiving_fails() -> None: +@pytest.mark.parametrize( + 'breaks', ['its_work_directory', 'packing_time', 'its_sandbox', 'the_provider_lifetime'] +) +def test_an_agent_that_breaks_its_archive_or_sandbox_scores_zero(breaks: str) -> None: + """A failed or overlong pack is taken as the agent's (its work directory moved, a link, + too big; the tools failing themselves can't be told apart), and only the agent's script + runs in the sandbox: masking any of these would let it dodge its 0.""" box = _box() real_exec = box.exec - box.exec = lambda command, **kw: ( # type: ignore[method-assign] - real_exec(command, **kw)._replace(exit_code=2) - if '-czf' in command - else real_exec(command, **kw) - ) - with pytest.raises(SandboxError, match='archive'): - run_agent(box, HarborTask.from_dir(HELLO), '') + + def exec_(command: str, **kwargs: object) -> object: + if 'agent.sh' in command and breaks == 'its_sandbox': + raise SandboxDiedError('the sandbox ran out of memory') + if 'agent.sh' in command and breaks == 'the_provider_lifetime': + time.sleep(0.3) # the script stalls until the provider retires the sandbox + raise SandboxExpiredError('the sandbox reached its provider lifetime') + if '-czf' in command and breaks == 'packing_time': + raise SandboxTimeoutError('command exceeded 600s') + if '-czf' in command: + return real_exec(command, **kwargs)._replace(exit_code=1) + return real_exec(command, **kwargs) + + box.exec = exec_ # type: ignore[method-assign] + result = run_agent(box, HarborTask.from_dir(HELLO), '') + + assert (result.score, result.has_verdict) == (0.0, True) + assert result.error_type == 'policy_sandbox_failure' assert box.deleted diff --git a/deepgym/tests/test_integrations_verl_tool.py b/deepgym/tests/test_integrations_verl_tool.py index 4f12b43..8a4dc09 100644 --- a/deepgym/tests/test_integrations_verl_tool.py +++ b/deepgym/tests/test_integrations_verl_tool.py @@ -9,6 +9,7 @@ import asyncio import importlib import logging +import re import sys import types from typing import Any @@ -17,7 +18,8 @@ from deepgym.exceptions import DeepGymError, SandboxError from deepgym.exceptions import TimeoutError as DeepGymTimeout -from deepgym.models import MultiTurnEnvironment +from deepgym.models import MODEL_FAULT_ERRORS, MultiTurnEnvironment +from deepgym.sandboxes import SandboxTimeoutError from deepgym.session import LocalBackend, Session WRITES_HELLO = ( @@ -316,26 +318,53 @@ def test_an_action_the_policy_broke_is_scored( assert masked == [False] +class _RefusedError(Exception): + """A provider refusing our credentials, as its SDK reports it.""" + + status_code = 401 + + @pytest.mark.parametrize( - ('failure', 'kept_out'), + ('failure', 'error_type'), [ - (ConnectionError('provider unavailable'), True), - (SandboxError('Sandbox local-session is not running'), False), # the policy killed it - (DeepGymTimeout('Verifier exceeded 30s'), False), # the policy left it hanging + (lambda mark: ConnectionError('provider unavailable'), 'unknown_error'), + (lambda mark: _RefusedError('HTTP 401'), 'unknown_error'), + # It timed out before it printed the consulted mark: it never read the policy's output. + ( + lambda mark: SandboxTimeoutError('exceeded 30s', f'\n{mark}:start\nloading fixtures'), + 'verifier_error', + ), + # Nothing it printed came back, so nothing shows it hadn't read the policy's output. + (lambda mark: SandboxTimeoutError('exceeded 30s'), 'policy_sandbox_failure'), + (lambda mark: SandboxTimeoutError('exceeded 30s', f'\n{mark}\n'), 'solution_timeout'), + ( + lambda mark: SandboxError('Sandbox local-session is not running'), + 'policy_sandbox_failure', + ), + (lambda mark: DeepGymTimeout('Verifier exceeded 30s'), 'policy_sandbox_failure'), + ], + ids=[ + 'provider_outage', + 'auth', + 'timed_out_before_reading', + 'timed_out_with_output_lost', + 'timed_out_reading', + 'sandbox_broken', + 'sandbox_hung', ], - ids=['provider_outage', 'sandbox_killed', 'sandbox_hung'], ) def test_a_final_verifier_that_cannot_run( verl: types.ModuleType, monkeypatch: pytest.MonkeyPatch, masked: list[bool], - failure: Exception, - kept_out: bool, + failure: Any, + error_type: str, ) -> None: - """Only a provider failure keeps the rollout out; a sandbox the policy broke scores 0.""" + """Only a failure that is ours for certain keeps the rollout out; the rest scores 0.""" + kept_out = error_type not in MODEL_FAULT_ERRORS def verify(backend: LocalBackend, code: str, timeout: int, env: Any = None) -> Any: - raise failure + raise failure(re.search(r'DEEPGYM_CONSULTED"\] = \'(\w+)\'', code)[1]) monkeypatch.setattr(LocalBackend, 'verify', verify) real_finish = verl.DeepGymSessionTool.finish @@ -353,7 +382,7 @@ async def finish(tool: Any, episode: str, *, score: bool) -> Any: assert output.extra_fields['reward_extra_info']['error'] == float(kept_out) assert output.response_mask == ([0, 0, 0] if kept_out else [1, 1, 1]) assert results[0].has_verdict is not kept_out - assert results[0].error_type == ('unknown_error' if kept_out else 'policy_sandbox_failure') + assert results[0].error_type == error_type assert masked == [kept_out] diff --git a/deepgym/tests/test_isolation.py b/deepgym/tests/test_isolation.py index 3c2a5ee..7d7aef7 100644 --- a/deepgym/tests/test_isolation.py +++ b/deepgym/tests/test_isolation.py @@ -4,6 +4,7 @@ and are skipped where neither works. """ +import contextlib import errno import json import os @@ -13,6 +14,7 @@ import sys import tempfile import time +from collections.abc import Iterator from pathlib import Path import pytest @@ -200,6 +202,29 @@ def test_no_start_signal_goes_through_sudo(monkeypatch: pytest.MonkeyPatch) -> N assert (prefix, fds, started()) == ([], (), True) # sudo would close the pipe +def test_the_consulted_mark_is_out_before_the_solution_is_released( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A solution that kills its sandbox the moment it starts can't beat the mark out.""" + from deepgym import solution_runtime + + script = tmp_path / 'solution.py' + script.write_text('print("hi")\n', encoding='utf-8') + events: list[str] = [] + real_signal = solution_runtime._start_signal + + @contextlib.contextmanager + def recorded_signal() -> Iterator[tuple]: + with real_signal() as (prefix, fds, started, release): + yield prefix, fds, started, lambda: (events.append('release'), release()) + + monkeypatch.setattr(solution_runtime, '_start_signal', recorded_signal) + monkeypatch.setattr(solution_runtime, '_announce', lambda: events.append('announce')) + + assert solution_runtime.run_solution(str(script)).stdout == 'hi\n' + assert events == ['announce', 'release'] + + def test_a_reader_that_cannot_start_is_a_launch_failure( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: diff --git a/deepgym/tests/test_sandboxes.py b/deepgym/tests/test_sandboxes.py index 20f1a2c..3a0a7ae 100644 --- a/deepgym/tests/test_sandboxes.py +++ b/deepgym/tests/test_sandboxes.py @@ -18,12 +18,14 @@ from collections.abc import Mapping from pathlib import Path +import httpx import pytest from deepgym import ComputerUseEnvironment, DeepGym from deepgym.exceptions import ( DeepGymError, SandboxError, + SandboxExpiredError, VerifierError, is_infrastructure_failure, ) @@ -42,13 +44,14 @@ SandboxImage, SandboxTimeoutError, open_sandbox, + protocol, sealed_path, ) from deepgym.sandboxes.apple import AppleContainer from deepgym.sandboxes.cli import DEFAULT_IMAGE from deepgym.sandboxes.daytona import cleanup_sandbox, snapshot_digest from deepgym.sandboxes.docker import DockerContainer -from deepgym.session import _SEALED_PATH_CHECK, SandboxBackend +from deepgym.session import _SEALED_PATH_CHECK, SandboxBackend, Session # Fake sandboxes pass session setup's check that only root can change TRUSTED_PATH. _SEALED_NEEDLE = 'root would run what others can change' @@ -615,6 +618,30 @@ def test_session_write_file_is_staged_then_written_as_agent() -> None: assert f'< {staged}' in box.commands[-1] and '/tmp/deepgym/a/b.py' in box.commands[-1] +@pytest.mark.parametrize('status', [401, 403]) +def test_a_step_verifier_upload_the_provider_refuses_has_no_verdict(status: int) -> None: + """Our credentials failing while we upload trusted verifier code is never the agent's.""" + box = FakeSandbox(network_blocked=True, answers=[_USERS]) + env = MultiTurnEnvironment(task='t', max_steps=3, step_verifier_code='print(1.0)') + session = Session(env, SandboxBackend(box)) + session.start() + request = httpx.Request('PUT', 'https://provider.test/files') + refused = httpx.HTTPStatusError( + 'refused', request=request, response=httpx.Response(status, request=request) + ) + + def upload(content: bytes, path: str) -> None: + if path.startswith('/opt/deepgym-verifier/'): + raise refused + + box.upload = upload # type: ignore[method-assign] + with pytest.raises(VerifierError, match='Could not upload the verifier'): + session.step(Action(action_type='bash', content='echo hi')) + + result = session.score() + assert session.done and (result.score, result.has_verdict) == (0.0, False) + + @pytest.mark.parametrize('ids', ['0 997 996 996', '997 997 996 0'], ids=['agent', 'group']) def test_session_refuses_root_ids_for_its_users(ids: str) -> None: output = f'{ids} 0 - - /usr/bin/setpriv /usr/bin/nsenter /usr/bin/sudo\n/tmp/deepgym' @@ -1156,6 +1183,190 @@ def test_daytona_sandbox_that_dies(phase: str) -> None: assert _judged(lambda: run_judge_in_sandbox(box, _JOB)) == 'no_verdict' +def _past_lifetime(monkeypatch: pytest.MonkeyPatch, lifetime: int) -> None: + """Move the clock that decides a stopped sandbox's cause past *lifetime* seconds.""" + real = time.monotonic + monkeypatch.setattr( + protocol, 'time', types.SimpleNamespace(monotonic=lambda: real() + lifetime) + ) + + +def test_e2b_sandbox_stopped_past_its_lifetime_expired_rather_than_died( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """E2B kills every sandbox an hour after creation, whatever runs in it: ours, no verdict.""" + + class NotRunningError(Exception): + pass + + def gone(*args: object, **kwargs: object) -> None: + raise NotRunningError('the sandbox was not found') + + class Client: + sandbox_id = 'e2b-old' + commands = types.SimpleNamespace(run=gone) + files = types.SimpleNamespace(write=gone) + + @staticmethod + def create(**kwargs: object) -> 'Client': + return Client() + + _install(monkeypatch, 'e2b', { + 'Sandbox': Client, + 'CommandExitException': type('ExitError', (Exception,), {}), + 'TimeoutException': type('SlowError', (Exception,), {}), + 'SandboxNotRunningException': NotRunningError, + 'SandboxUnreachableException': type('UnreachableError', (Exception,), {}), + }) # fmt: skip + monkeypatch.delitem(sys.modules, 'deepgym.sandboxes.e2b.sandbox', raising=False) + module = importlib.import_module('deepgym.sandboxes.e2b.sandbox') + box = module.E2BSandbox() + calls = (lambda: box.exec('ls', timeout=5), lambda: box.upload(b'x', '/opt/x')) + + for call in calls: + with pytest.raises(SandboxDiedError): + call() + _past_lifetime(monkeypatch, module.LIFETIME_S) + for call in calls: + with pytest.raises(SandboxExpiredError, match='lifetime'): + call() + + +@pytest.mark.parametrize('report', ['terminated', 'failed_exit']) +def test_modal_sandbox_stopped_past_its_lifetime_expired_rather_than_died( + monkeypatch: pytest.MonkeyPatch, report: str +) -> None: + box = _dying_modal_box(monkeypatch, 'gone', report, '') + _past_lifetime(monkeypatch, sys.modules['deepgym.sandboxes.modal.sandbox'].LIFETIME_S) + + with pytest.raises(SandboxExpiredError, match='lifetime'): + box.exec('gone') + + +def test_a_daytona_timeout_keeps_what_the_command_printed( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The sandbox enforces the deadline, so the consulted mark printed first comes back.""" + pytest.importorskip('daytona') + from deepgym.sandboxes.daytona import DaytonaSandbox + + box = _dying_daytona_box('never runs', RuntimeError('unused')) + calls: list[tuple[str, object]] = [] + real_exec = box.process.exec + + def exec_(command: str, **kwargs: object) -> types.SimpleNamespace: + calls.append((command, kwargs['timeout'])) + if 'slow' in command: + return types.SimpleNamespace(exit_code=124, result='printed first\n') + return real_exec(command, **kwargs) + + box.process = types.SimpleNamespace(exec=exec_) + daytona = types.SimpleNamespace(create=lambda params, timeout: box, delete=print) + sandbox = DaytonaSandbox(daytona, Environment(task='t', verifier_code='return 1.0')) + clock = iter([0.0, 10.0]) + monkeypatch.setattr('deepgym.sandboxes.daytona.sandbox.time.monotonic', lambda: next(clock)) + + with pytest.raises(SandboxTimeoutError) as raised: + sandbox.exec('slow', timeout=5) + + assert raised.value.output == 'printed first\n' + command, waited = calls[-1] + assert '/usr/bin/timeout --kill-after=2 5 /bin/sh -c slow' in command and waited > 5 + + +def test_the_consulted_mark_is_in_the_private_script_never_on_a_command_line() -> None: + """Any process in the sandbox can list command lines; only the verifier user reads its + script. So the agent never learns the mark, and only the harness prints it.""" + box = FakeSandbox(network_blocked=True, answers=[_USERS]) + scripts: list[bytes] = [] + box.upload = lambda content, path: scripts.append(content) # type: ignore[method-assign] + env = MultiTurnEnvironment(task='t', final_verifier_code='print(1.0)') + session = Session(env, SandboxBackend(box)) + session.start() + with contextlib.suppress(VerifierError): + session.score() + + mark = re.search(rb'DEEPGYM_CONSULTED"\] = \'([0-9a-f]{32})\'', scripts[-1]) + assert mark is not None + assert not any(mark[1].decode() in command for command in box.commands) + assert not any('DEEPGYM_CONSULTED' in command for command in box.commands) + + +def test_only_uploading_the_verifier_is_ours_whatever_the_provider_says() -> None: + """Its code to its own folder is ours alone. Its launch is not: it runs in the agent's + work directory, which the agent can move away.""" + box = FakeSandbox(network_blocked=True, answers=[_USERS]) + backend = SandboxBackend(box) + real_upload, real_exec = box.upload, box.exec + + def upload(content: bytes, path: str) -> None: + raise _ProviderError('HTTP 400: bad request') + + box.upload = upload # type: ignore[method-assign] + with pytest.raises(VerifierError, match='Could not upload the verifier'): + backend.verify('print(1)', timeout=5) + + def exec_(command: str, **kwargs: object) -> ExecResult: + if '/opt/deepgym-verifier/' in command: + raise _ProviderError('HTTP 400: no such working directory') + return real_exec(command, **kwargs) + + box.upload, box.exec = real_upload, exec_ # type: ignore[method-assign] + with pytest.raises(_ProviderError): + backend.verify('print(1)', timeout=5) + + # Docker and Apple report an ordinary failed upload as a SandboxError: still ours. A + # sandbox found stopped is not. + for error, raised in ( + (SandboxError('Failed to upload: no space left'), VerifierError), + (SandboxDiedError('the container stopped'), SandboxDiedError), + ): + + def upload_fails(content: bytes, path: str, error: Exception = error) -> None: + raise error + + box.upload = upload_fails # type: ignore[method-assign] + with pytest.raises(raised): + backend.verify('print(1)', timeout=5) + + +class _ProviderError(Exception): + """An SDK error carrying an HTTP status, as Daytona's do.""" + + status_code = 400 + + +def test_daytona_upload_to_a_stopped_sandbox_is_a_death() -> None: + daytona_sdk = pytest.importorskip('daytona') + from deepgym.sandboxes.daytona import DaytonaSandbox + + stopping = _dying_daytona_box('never runs', daytona_sdk.DaytonaError('unused')) + + def upload_file(data: bytes, path: str) -> None: + stopping.state = 'stopped' + raise daytona_sdk.DaytonaError('sandbox is not running') + + stopping.fs = types.SimpleNamespace(upload_file=upload_file) + daytona = types.SimpleNamespace(create=lambda params, timeout: stopping, delete=print) + box = DaytonaSandbox(daytona, Environment(task='t', verifier_code='return 1.0')) + + with pytest.raises(SandboxDiedError): + box.upload(b'x', '/opt/x') + + +def test_cli_upload_to_a_stopped_container_is_a_death(monkeypatch: pytest.MonkeyPatch) -> None: + def fake_run(argv: list[str], **kwargs: object) -> subprocess.CompletedProcess: + code = 1 if argv[1:3] == ['exec', '--interactive'] else 0 + return subprocess.CompletedProcess(argv, code, b'', b'no such container') + + monkeypatch.setattr(subprocess, 'run', fake_run) + box = DockerContainer() + monkeypatch.setattr(box, 'running', lambda: []) + + with pytest.raises(SandboxDiedError): + box.upload(b'x', '/opt/x') + + @pytest.mark.parametrize('reported_by', ['the_error', 'a_queue_timeout', 'the_sandbox_record']) def test_daytona_evicting_a_sandbox_is_not_the_solutions_doing(reported_by: str) -> None: daytona_sdk = pytest.importorskip('daytona') @@ -1261,7 +1472,9 @@ def test_daytona_desktop_runs_as_the_agent_and_is_locked_down( before = ' '.join(events[:started]) assert '/usr/local/bin/xfce4-session' in before, 'the desktop must not run as the sudoer' prepare = next(c for c in events[:started] if '/usr/local/bin/xfce4-session' in c) - assert prepare.startswith("/usr/bin/sudo -n -E /bin/sh -c '/usr/bin/env -i ") + assert ( + prepare.startswith('/usr/bin/sudo -n -E /bin/sh -c ') and '/usr/bin/env -i -- ' in prepare + ) assert '--reuid=997' in prepare and '/opt/deepgym-netns' in prepare for blocked in ('thunar-uca', 'Thunar/uca.xml', 'gio-launch-desktop', 'xfce4-terminal', 'idle'): assert blocked in prepare, f'{blocked} must be locked down before the desktop starts' diff --git a/deepgym/tests/test_session.py b/deepgym/tests/test_session.py index 6ef0851..6f693af 100644 --- a/deepgym/tests/test_session.py +++ b/deepgym/tests/test_session.py @@ -11,10 +11,17 @@ from pydantic import ValidationError from deepgym import AsyncDeepGym, ComputerUseEnvironment, DeepGym -from deepgym.exceptions import ActionError, DeepGymError, SandboxError, VerifierError +from deepgym import session as session_module +from deepgym.exceptions import ( + ActionError, + DeepGymError, + SandboxError, + SandboxExpiredError, + VerifierError, +) from deepgym.models import Action, MultiTurnEnvironment, Observation, Trajectory -from deepgym.sandboxes import SandboxDiedError -from deepgym.session import Session, _signals_done, run_episode +from deepgym.sandboxes import SandboxDiedError, SandboxTimeoutError +from deepgym.session import Session, _signals_done, failure_result, run_episode PASS_VERIFIER = ( 'import json\n' @@ -325,8 +332,12 @@ def test_a_write_the_agents_own_files_block_is_rejected_and_the_episode_goes_on( @pytest.mark.parametrize( 'blocker', - ['mkdir _step_output.txt', 'ln -s missing_dir/output _step_output.txt'], - ids=['folder', 'dangling_link'], + [ + 'mkdir _step_output.txt', + 'ln -s missing_dir/output _step_output.txt', + 'mkfifo _step_output.txt', + ], + ids=['folder', 'dangling_link', 'fifo'], ) def test_blocking_the_step_output_ends_the_episode_with_a_judged_zero( self, dg: DeepGym, blocker: str @@ -403,9 +414,12 @@ def test_run_episode_returns_the_result_a_failed_step_fixed(self, dg: DeepGym) - assert len(trajectory.steps) == 1 # the agent blocked the harness: its own 0 assert (result.has_verdict, result.error_type) == (True, 'policy_sandbox_failure') + @pytest.mark.parametrize('code', [errno.EIO, errno.ENOSPC, errno.EDQUOT]) def test_a_step_output_write_that_fails_is_the_agents_zero( - self, dg: DeepGym, monkeypatch: pytest.MonkeyPatch + self, dg: DeepGym, monkeypatch: pytest.MonkeyPatch, code: int ) -> None: + """The agent can fill the disk, or plant a link the write follows, and free space + again before anything checks: no write failure after it acted is ours for certain.""" env = MultiTurnEnvironment( task='t', max_steps=3, @@ -416,20 +430,333 @@ def test_a_step_output_write_that_fails_is_the_agents_zero( def disk_error(path: object, *args: object, **kwargs: object) -> int: if str(path).endswith('_step_output.txt'): - raise OSError(errno.EIO, 'Input/output error') + raise OSError(code, os.strerror(code)) return real_open(path, *args, **kwargs) with dg.session(env) as session: session.start() monkeypatch.setattr(os, 'open', disk_error) - with pytest.raises(SandboxError, match='Input/output error'): + with pytest.raises(SandboxError, match=os.strerror(code)): session.step(Action(action_type='bash', content='echo harmless')) monkeypatch.undo() result = session.score() - # A failed write with no provider failure behind it follows from the agent's actions. assert session.done and len(session.trajectory.steps) == 1 assert (result.has_verdict, result.error_type) == (True, 'policy_sandbox_failure') + def test_a_verifier_the_agents_output_hangs_is_its_zero( + self, dg: DeepGym, monkeypatch: pytest.MonkeyPatch + ) -> None: + """A verifier stuck reading what the agent made (a FIFO) timed out on the agent.""" + monkeypatch.setattr('deepgym.session.DEADLINE_GRACE_S', 0) + reads = 'print(float(open("answer.txt").read() == "42\\n"))\n' + env = MultiTurnEnvironment( + task='t', + max_steps=3, + timeout_per_step=1, + step_verifier_code=reads, + final_verifier_code=reads, + ) + with dg.session(env) as session: + session.start() + answered = session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + hung = session.step( + Action(action_type='bash', content='rm answer.txt; mkfifo answer.txt') + ) + verdict = session.step_verdict + result = session.score() + + assert answered.reward == 1.0 # the plain-float form still parses + assert (hung.reward, hung.done, verdict['error_type']) == (0.0, False, 'solution_timeout') + assert (result.score, result.has_verdict, result.error_type) == ( + 0.0, + True, + 'solution_timeout', + ) + + def test_a_verifier_that_hangs_before_reading_the_agents_output_has_no_verdict( + self, dg: DeepGym, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr('deepgym.session.DEADLINE_GRACE_S', 0) + hangs = 'import time\ntime.sleep(30)\n' + step_env = MultiTurnEnvironment( + task='t', max_steps=3, timeout_per_step=1, step_verifier_code=hangs + ) + final_env = MultiTurnEnvironment( + task='t', max_steps=3, timeout_per_step=1, final_verifier_code=hangs + ) + with dg.session(step_env) as session: + session.start() + with pytest.raises(VerifierError, match='timed out'): + session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + result = session.score() + with dg.session(final_env) as session: + session.start() + session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + with pytest.raises(VerifierError, match='before it read'): + session.score() + + assert (result.has_verdict, result.error_type) == (False, 'verifier_error') + + @pytest.mark.parametrize( + 'consults', + [ + 'import os\nos.listdir(".")\n', + 'import glob\nglob.glob("**/*.txt", recursive=True)\n', + 'import subprocess\nsubprocess.run(["true"])\n', + 'import os, subprocess\n' + 'subprocess.run(["cat", os.path.abspath("answer.txt")], cwd="/", stdout=-3)\n', + 'import contextlib, io\nwith contextlib.redirect_stdout(io.StringIO()):\n' + ' open("answer.txt").close()\n', + ], + ids=[ + 'lists_its_folder', + 'globs_its_folder', + 'runs_a_program_there', + 'runs_a_program_on_its_file', + 'captures_stdout', + ], + ) + def test_a_verifier_can_not_be_kept_from_marking_what_it_consulted( + self, dg: DeepGym, monkeypatch: pytest.MonkeyPatch, consults: str + ) -> None: + """Listing the agent's folder or running a program in it consults the agent, and + capturing stdout can't hide the mark: a hang after that is the agent's 0.""" + monkeypatch.setattr('deepgym.session.DEADLINE_GRACE_S', 0) + hangs = consults + 'import time\ntime.sleep(30)\n' + env = MultiTurnEnvironment( + task='t', max_steps=2, timeout_per_step=1, final_verifier_code=hangs + ) + with dg.session(env) as session: + session.start() + session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + result = session.score() + + assert (result.score, result.has_verdict, result.error_type) == ( + 0.0, + True, + 'solution_timeout', + ) + + @pytest.mark.parametrize('stalls', [True, False], ids=['agent_stalls', 'host_pauses']) + def test_a_provider_lifetime_the_agent_ran_out_is_its_zero( + self, dg: DeepGym, monkeypatch: pytest.MonkeyPatch, stalls: bool + ) -> None: + """An expiry is ours only when the agent's own actions didn't take most of the time.""" + env = MultiTurnEnvironment(task='t', max_steps=2, final_verifier_code=PASS_VERIFIER) + + def expired(*args: object, **kwargs: object) -> None: + raise SandboxExpiredError('the sandbox reached its provider lifetime') + + with dg.session(env) as session: + session.start() + session.step(Action(action_type='bash', content='sleep 0.6' if stalls else 'true')) + if not stalls: + time.sleep(0.6) # the host, thinking between steps + monkeypatch.setattr(session._backend, 'verify', expired) + if stalls: + result = session.score() + assert (result.has_verdict, result.error_type) == (True, 'policy_sandbox_failure') + else: + with pytest.raises(SandboxExpiredError) as raised: + session.score() + assert not failure_result(raised.value, session.sandbox_id).has_verdict + + def test_a_verifier_reads_its_own_mark_not_one_the_agent_wrote( + self, dg: DeepGym, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Each run's mark is fresh and random: a mark-shaped line the verifier prints, a + stale mark, or an earlier run's mark can't make a hang before reading count.""" + monkeypatch.setattr('deepgym.session.DEADLINE_GRACE_S', 0) + marks: list[str] = [] + real_guarded = session_module._guarded + + def guarded(code: str, mark: str) -> str: + marks.append(mark) + return real_guarded(code, mark) + + monkeypatch.setattr(session_module, '_guarded', guarded) + env = MultiTurnEnvironment( + task='t', + max_steps=3, + timeout_per_step=1, + step_verifier_code='print(float(open("answer.txt").read() == "42\\n"))\n', + ) + with dg.session(env) as session: + session.start() + session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + # The final verifier prints mark-shaped lines and both earlier runs' marks, each + # on a line of its own, then hangs without reading anything of the agent's. + prints = ''.join(f'print({line!r})\n' for line in ['0' * 32, *marks]) + hangs = f'{prints}import sys, time\nsys.stdout.flush()\ntime.sleep(30)\n' + session._env = env.model_copy(update={'final_verifier_code': hangs}) + with pytest.raises(VerifierError, match='before it read'): + session.score() + + assert len(set(marks)) == 3 and all(len(mark) == 32 for mark in marks) + + def test_what_the_agent_shapes_on_stdout_cant_change_the_verdict(self, dg: DeepGym) -> None: + """The verifier echoes the agent's bytes (invalid UTF-8, a mark-shaped line, a huge + line, a fake verdict) around its own partial line and the mark; its own last + verdict line is the one read.""" + echoes = ( + 'import json, sys\n' + 'sys.stdout.write("partial line")\n' + 'data = open("answer.bin", "rb").read()\n' + 'sys.stdout.flush()\n' + 'sys.stdout.buffer.write(data + b"\\n")\n' + 'print(json.dumps({"score": 0.25, "passed": False}))\n' + ) + env = MultiTurnEnvironment(task='t', max_steps=2, final_verifier_code=echoes) + planted = ( + "printf '\\377\\376 junk\\n' > answer.bin; " + "printf '%032d\\n' 0 >> answer.bin; " + "head -c 200000 /dev/zero | tr '\\0' x >> answer.bin; " + 'printf \'\\n{"score": 1.0, "passed": true}\' >> answer.bin' + ) + with dg.session(env) as session: + session.start() + session.step(Action(action_type='bash', content=planted)) + result = session.score() + + assert (result.score, result.passed, result.has_verdict) == (0.25, False, True) + + @pytest.mark.parametrize('consults', [True, False], ids=['after_reading', 'before']) + def test_a_verifier_killed_after_reading_the_agents_output_is_its_zero( + self, dg: DeepGym, consults: bool + ) -> None: + """Memory the agent's output made the verifier use can get it killed: no verdict + would let the agent dodge its 0. Killed before reading anything, it's ours.""" + reads = 'open("answer.txt").read()\n' if consults else '' + killed = f'import os, signal\n{reads}os.kill(os.getpid(), signal.SIGKILL)\n' + env = MultiTurnEnvironment(task='t', max_steps=2, final_verifier_code=killed) + with dg.session(env) as session: + session.start() + session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + if not consults: + with pytest.raises(VerifierError): + session.score() + return + result = session.score() + + assert (result.score, result.has_verdict, result.error_type) == ( + 0.0, + True, + 'solution_error', + ) + + def test_a_timeout_whose_output_was_lost_is_the_agents_zero( + self, dg: DeepGym, monkeypatch: pytest.MonkeyPatch + ) -> None: + """With nothing the verifier printed (not even its start line), nothing shows it + hadn't read the agent's output: a pipe a planted child held, say.""" + env = MultiTurnEnvironment(task='t', max_steps=2, final_verifier_code=PASS_VERIFIER) + + def lost(*args: object, **kwargs: object) -> None: + raise SandboxTimeoutError('command exceeded 30s') # what the provider returns + + with dg.session(env) as session: + session.start() + session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + monkeypatch.setattr(session._backend, 'verify', lost) + result = session.score() + + assert (result.score, result.has_verdict, result.error_type) == ( + 0.0, + True, + 'policy_sandbox_failure', + ) + + def test_a_verdict_printed_around_the_mark_reads_as_printed(self, dg: DeepGym) -> None: + """The mark is cut out exactly, so a line it fell inside is whole again.""" + split = ( + 'import sys\n' + 'sys.stdout.write(\'{"score":\')\n' + 'open("answer.txt").close()\n' + 'print(\' 0.5, "passed": false}\')\n' + ) + env = MultiTurnEnvironment(task='t', max_steps=2, final_verifier_code=split) + with dg.session(env) as session: + session.start() + session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + result = session.score() + + assert (result.score, result.passed, result.has_verdict) == (0.5, False, True) + + @pytest.mark.parametrize( + 'fails', + [ + 'import sys\nsys.exit(2)\n', + 'print(\'{"score": 5}\')\n', + 'print("not a verdict")\n', + 'print(\'{"score": 0, "passed": false, "error_type": "runtime_error"}\')\n' + 'import sys\nsys.exit(2)\n', + ], + ids=['exits_silently', 'invalid_score', 'no_verdict_line', 'reports_its_own_error'], + ) + def test_a_verifier_that_fails_after_reading_the_agents_output_is_its_zero( + self, dg: DeepGym, fails: str + ) -> None: + """Whatever the agent's output made the verifier do, no verdict would let it dodge.""" + reads = 'open("answer.txt").read()\n' + env = MultiTurnEnvironment( + task='t', + max_steps=3, + step_verifier_code=reads + fails, + final_verifier_code=reads + fails, + ) + with dg.session(env) as session: + session.start() + step = session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + verdict = session.step_verdict + result = session.score() + + assert (step.reward, verdict['error_type']) == (0.0, 'solution_error') + assert (result.score, result.has_verdict, result.error_type) == ( + 0.0, + True, + 'solution_error', + ) + + def test_the_agents_code_failing_to_launch_has_no_verdict_even_after_reading( + self, dg: DeepGym + ) -> None: + """The crash guard's launch failure is the machine's, whatever the verifier read.""" + launch = ( + 'open("answer.txt").read()\n' + 'print(\'{"score": 0, "passed": false, "error_type": "sandbox_error"}\')\n' + 'import sys\nsys.exit(2)\n' + ) + env = MultiTurnEnvironment(task='t', max_steps=2, final_verifier_code=launch) + with dg.session(env) as session: + session.start() + session.step(Action(action_type='bash', content='echo 42 > answer.txt')) + result = session.score() + + assert (result.has_verdict, result.error_type) == (False, 'sandbox_error') + + def test_a_failure_the_rule_cant_read_is_the_agents_zero(self) -> None: + """An exception that breaks the classifier is never a reason to mask.""" + + class UnreadableError(Exception): + @property + def status_code(self) -> int: + raise RuntimeError('broken SDK error') + + def __str__(self) -> str: + raise RuntimeError('no message either') + + result = failure_result(UnreadableError(), 'sb-1') + + assert (result.score, result.has_verdict) == (0.0, True) + assert result.error_type == 'policy_sandbox_failure' + + def test_output_that_isnt_utf8_is_shown_not_a_failure(self, dg: DeepGym) -> None: + with dg.session(MultiTurnEnvironment(task='t', max_steps=2)) as session: + session.start() + obs = session.step(Action(action_type='bash', content="printf '\\377ok'")).observation + assert obs.content == '\ufffdok' and not session.done + def test_step_verifier_model_fault_is_a_reward(self, dg: DeepGym) -> None: model_fault = 'print(\'{"score": 0.0, "passed": false, "error_type": "invalid_patch"}\')\n' env = MultiTurnEnvironment(task='t', max_steps=1, step_verifier_code=model_fault)