Repository navigation
fix: sessions give no verdict only for failures that are ours for certain - #78
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Once the agent has acted in a session, a failure is its judged 0 (
policy_sandbox_failure) unless it is ours for certain. The rule isdeepgym.session.failure_result, shared by sessions, verl'sdeepgym_agentloop and Harbor's agent phase.No verdict after the agent acted:
is_infrastructure_failure: transport errors, provider call timeouts, 5xx, 429, eviction, and now 401/403VerifierError: the verifier's code upload failed, or it failed or timed out before printing the consulted marksandbox_error)SandboxExpiredError(E2B/Modal lifetime), unless the agent's own actions took at least half the elapsed timeEverything else is the agent's 0. That includes a failed harness write (EIO, ENOSPC, EDQUOT), a verifier that is killed or gives no verdict after the mark (
solution_error), and an exception the classifier can't read.Changes
solution_timeout. A timeout whose output was lost (no start line) is a judged 0.SandboxExpiredError. E2B, Daytona and Docker/Apple uploads to a stopped sandbox raiseSandboxDiedError.run_solutionprints the consulted mark before releasing the solution.Tests: regressions for each review finding (401/403 on verifier upload, verl final-verifier timeout, E2B/Modal expiry, step-verifier timeout after the mark), plus each no-verdict path the agent might trigger. Affected files pass locally (1070 tests). A live Daytona run covered a pass, a timeout after the mark (
solution_timeout) and a timeout before it (no verdict); all sandboxes were deleted.Known gaps, not addressed here
os.stat, or a symlink resolved before opening, prints no mark. Fd-level capture of stdout hides it.Session.score()still raises (not classifies) for failures before the mark, andrun_episodepasses that on. Harbor's grader phase is not classified._extract_json_linesplits on U+2028 and similar, so a verdict line echoing such characters is unparseable.Need help on this PR? Tag
@codesmith-botwith what you need.