Skip to content

fix: sessions give no verdict only for failures that are ours for certain - #78

Merged
abhishekgahlot2 merged 2 commits into
mainfrom
fix/session-attribution
Oct 14, 2026
Merged

abhishekgahlot2 merged 2 commits into
mainfrom
fix/session-attribution

Conversation

@abhishekgahlot2

@abhishekgahlot2 abhishekgahlot2 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Once the agent has acted in a session, a failure is its judged 0 (policy_sandbox_failure) unless it is ours for certain. The rule is deepgym.session.failure_result, shared by sessions, verl's deepgym_agent loop and Harbor's agent phase.

No verdict after the agent acted:

  • is_infrastructure_failure: transport errors, provider call timeouts, 5xx, 429, eviction, and now 401/403
  • VerifierError: the verifier's code upload failed, or it failed or timed out before printing the consulted mark
  • a launch failure of the agent's code reported by the crash guard (sandbox_error)
  • SandboxExpiredError (E2B/Modal lifetime), unless the agent's own actions took at least half the elapsed time
  • interrupts

Everything else is the agent's 0. That includes a failed harness write (EIO, ENOSPC, EDQUOT), a verifier that is killed or gives no verdict after the mark (solution_error), and an exception the classifier can't read.

Changes

  • Session verifiers get a per-call consulted mark, embedded in their uploaded script rather than on a command line. It goes to fd 1, is cut from stdout exactly, and listing the agent's folder or starting a program in it or on its files counts as consulting. A timeout after the mark is solution_timeout. A timeout whose output was lost (no start line) is a judged 0.
  • Daytona enforces deadlines in the sandbox, so timed-out output (and the mark) comes back.
  • E2B/Modal report a stop after their lifetime as SandboxExpiredError. E2B, Daytona and Docker/Apple uploads to a stopped sandbox raise SandboxDiedError.
  • Harbor: a failed or overlong pack, a dead sandbox, or a lifetime spent mostly by the agent's script is the agent's 0. All of these used to give no verdict.
  • A FIFO at a harness write path is refused at once. Local output that isn't UTF-8 is shown, not an error.
  • run_solution prints the consulted mark before releasing the solution.

Tests: regressions for each review finding (401/403 on verifier upload, verl final-verifier timeout, E2B/Modal expiry, step-verifier timeout after the mark), plus each no-verdict path the agent might trigger. Affected files pass locally (1070 tests). A live Daytona run covered a pass, a timeout after the mark (solution_timeout) and a timeout before it (no verdict); all sandboxes were deleted.

Known gaps, not addressed here

  • Expiry is inferred from when a stop is observed, so a policy-caused death first seen after the lifetime counts as an expiry. Provider evidence of lifetime termination would fix it. The half-time rule also leaves a window; a per-episode action budget against the remaining lifetime would close it.
  • A consult only through os.stat, or a symlink resolved before opening, prints no mark. Fd-level capture of stdout hides it.
  • Session.score() still raises (not classifies) for failures before the mark, and run_episode passes that on. Harbor's grader phase is not classified.
  • Harbor archives have no expanded-size or file-count limit, and the agent script upload sits outside attribution.
  • The local backend's timeout drain can block on a detached child, and its verifier code (with the mark) is on a command line. The local backend is dev-only.
  • A genuine remote EIO on an upload or pack is scored as the agent's 0. Disk exhaustion that only breaks a verifier before it consults is still masked.
  • _extract_json_line splits on U+2028 and similar, so a verdict line echoing such characters is unparseable.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@abhishekgahlot2
abhishekgahlot2 merged commit bc73a5e into main Oct 14, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant